Federation F-ticks aggregator deployment
The federation F-ticks aggregator can be deployed as a central syslog hub that is configured to forward F-ticks received from the IdPs to the central eduGAIN F-ticks collector. There are many options for deploying federation F-ticks aggregator, starting from a simple syslog server, to a more robust and complex log management system, such as ELK stack.
Following are guidelines on how to deploy ELK stack as federation F-ticks aggregator.
- Install ELK stack from https://github.com/GEANT/FTICKS-ELK-Ansible
- Configure Logstash to listen to port 514
- Logstash can be configured to listen to ephemeral ports (eg. 1514/udp)
input { syslog { port => 1514 codec => plain syslog_field => "syslog" } }
- Configure persistent iptables rule to forward all udp traffic from port 514/udp to 1514/udp
iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to-port 1514
- Configure Logstash to process the received F-tick (for federation processing purposes)
filter{ mutate{ remove_tag => [ "_grokparsefailure_sysloginput" ] } grok{ match => [ "message", "(<%{POSINT:priority1}>%{SYSLOGTIMESTAMP:timestamp1} %{GREEDYDATA:hostname1})?<%{POSINT:priority}>%{SYSLOGTIMESTAMP:timestamp} %{GREEDYDATA:hostname} (%{GREEDYDATA:appname} )?\[%{GREEDYDATA:process}\](\:)? %{GREEDYDATA:syslog_message}" ] } if "_grokparsefailure" in [tags] { mutate{ remove_tag => [ "_grokparsefailure" ] add_tag => [ "not_ftick" ] } } grok{ match => [ "syslog_message", "(%{GREEDYDATA:bean} )?F-TICKS/%{GREEDYDATA:federation}/%{GREEDYDATA:version}[#]TS=%{GREEDYDATA:TS}[#]RP=%{GREEDYDATA:RP}[#]AP=%{GREEDYDATA:AP}([#]AM=%{GREEDYDATA:AM})?([#]RESULT=%{GREEDYDATA:RESULT})?[#]" ] } if "_grokparsefailure" in [tags] { mutate{ remove_tag => [ "_grokparsefailure" ] } grok{ match => [ "syslog_message", "(%{GREEDYDATA:bean} )?F-TICKS/%{GREEDYDATA:federation}/%{GREEDYDATA:version}[#]RESULT=%{GREEDYDATA:RESULT}[#]AP=%{GREEDYDATA:AP}[#]RP=%{GREEDYDATA:RP}[#]TS=%{GREEDYDATA:TS}[#]" ] } } grok { match => [ "RP", "(%{URIPROTO:RP_uri_proto}://(?:%{USER:RP_user}(?::[^@]*)?@)?)?(?:%{URIHOST:RP_uri_domain})?(?:%{URIPATHPARAM:RP_uri_param})?" ] } grok { match => [ "AP", "(%{URIPROTO:AP_uri_proto}://(?:%{USER:AP_user}(?::[^@]*)?@)?)?(?:%{URIHOST:AP_uri_domain})?(?:%{URIPATHPARAM:AP_uri_param})?" ] } geoip { source => "AP_uri_domain" target => "AP_geoip" } geoip { source => "RP_uri_domain" target => "RP_geoip" } }
- Configure Logstash to forward all F-ticks to the central GENAT aggregator
output { # output to central GEANT Fticks aggregator syslog { host => "collector.f-ticks.edugain.org" message => "%{syslog_message}" codec => "plain" port => 514 } # output to a local elasticsearch index for federation level visualizations elasticsearch { id => "fticks" hosts => [ "localhost:9200" ] index => "fticks_edugain" } }
- For better indexing of data, it is necessary to setup an index template in the elasticsearch configuration
$ wget https://raw.githubusercontent.com/GEANT/FTicks-ELK/master/haproxy/config/fticks.template $ wget https://raw.githubusercontent.com/GEANT/FTicks-ELK/master/haproxy/config/fticks.index $ curl -X PUT -H 'Content-Type: application/json' http://localhost:9200/_template/fticks_template -d '@fticks.template' $ curl -X PUT -H 'Content-Type: application/json' http://localhost:9200/fticks -d '@/fticks.index'
- Logstash can be configured to listen to ephemeral ports (eg. 1514/udp)