Federation F-ticks aggregator deployment

The federation F-ticks aggregator can be deployed as a central syslog hub that is configured to forward F-ticks received from the IdPs to the central eduGAIN F-ticks collector. There are many options for deploying federation F-ticks aggregator, starting from a simple syslog server, to a more robust and complex log management system, such as ELK stack.

Following are guidelines on how to deploy ELK stack as federation F-ticks aggregator.

• Install ELK stack from https://github.com/GEANT/FTICKS-ELK-Ansible
• Configure Logstash to listen to port 514
  • Logstash can be configured to listen to ephemeral ports (eg. 1514/udp)
    
    

    input {
      syslog {
        port => 1514
        codec => plain
        syslog_field => "syslog"
      }
    }
    

    
    
  • Configure persistent iptables rule to forward all udp traffic from port 514/udp to 1514/udp
    
    

    iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to-port 1514
    

    
    
  • Configure Logstash to process the received F-tick (for federation processing purposes)
    
    

     
    filter{
      mutate{
        remove_tag => [ "_grokparsefailure_sysloginput" ]
      }
    
      grok{
        match => [ "message", "(<%{POSINT:priority1}>%{SYSLOGTIMESTAMP:timestamp1} %{GREEDYDATA:hostname1})?<%{POSINT:priority}>%{SYSLOGTIMESTAMP:timestamp} %{GREEDYDATA:hostname} (%{GREEDYDATA:appname} )?\[%{GREEDYDATA:process}\](\:)? %{GREEDYDATA:syslog_message}" ]
      }
      if "_grokparsefailure" in [tags] {
        mutate{
          remove_tag => [ "_grokparsefailure" ]
          add_tag => [ "not_ftick" ]
        }
      }
    
      grok{
        match => [ "syslog_message", "(%{GREEDYDATA:bean} )?F-TICKS/%{GREEDYDATA:federation}/%{GREEDYDATA:version}[#]TS=%{GREEDYDATA:TS}[#]RP=%{GREEDYDATA:RP}[#]AP=%{GREEDYDATA:AP}([#]AM=%{GREEDYDATA:AM})?([#]RESULT=%{GREEDYDATA:RESULT})?[#]" ]
      }
      if "_grokparsefailure" in [tags] {
        mutate{
          remove_tag => [ "_grokparsefailure" ]
        }
        grok{
          match => [ "syslog_message", "(%{GREEDYDATA:bean} )?F-TICKS/%{GREEDYDATA:federation}/%{GREEDYDATA:version}[#]RESULT=%{GREEDYDATA:RESULT}[#]AP=%{GREEDYDATA:AP}[#]RP=%{GREEDYDATA:RP}[#]TS=%{GREEDYDATA:TS}[#]" ]
        }
      }
    
      grok {
         match => [ "RP", "(%{URIPROTO:RP_uri_proto}://(?:%{USER:RP_user}(?::[^@]*)?@)?)?(?:%{URIHOST:RP_uri_domain})?(?:%{URIPATHPARAM:RP_uri_param})?" ]
      }
    
      grok {
         match => [ "AP", "(%{URIPROTO:AP_uri_proto}://(?:%{USER:AP_user}(?::[^@]*)?@)?)?(?:%{URIHOST:AP_uri_domain})?(?:%{URIPATHPARAM:AP_uri_param})?" ]
      }
    
      geoip {
         source => "AP_uri_domain"
         target => "AP_geoip"
      }
    
      geoip {
         source => "RP_uri_domain"
         target => "RP_geoip"
      }
    }
    

    
    
  • Configure Logstash to forward all F-ticks to the central GENAT aggregator
    
    

    output {
    
        # output to central GEANT Fticks aggregator
        syslog {
          host => "collector.f-ticks.edugain.org"
          message => "%{syslog_message}"
          codec => "plain"
          port => 514
        }
    
        # output to a local elasticsearch index for federation level visualizations
        elasticsearch {
          id => "fticks"
          hosts => [ "localhost:9200" ]
          index => "fticks_edugain"
        }
    
    }
    

    
    
  • For better indexing of data, it is necessary to setup an index template in the elasticsearch configuration
    
    

    $ wget https://raw.githubusercontent.com/GEANT/FTicks-ELK/master/haproxy/config/fticks.template
    $ wget https://raw.githubusercontent.com/GEANT/FTicks-ELK/master/haproxy/config/fticks.index
    $ curl -X PUT -H 'Content-Type: application/json' http://localhost:9200/_template/fticks_template -d '@fticks.template'
    $ curl -X PUT -H 'Content-Type: application/json' http://localhost:9200/fticks -d '@/fticks.index'