F-Ticks Statistics ......... !!under construction!!

Join as Federation provider

How to deploy a national f-ticks aggregator

National Fticks aggregator can be deployed as a central syslog hub that is configured to forward federation Fticks towards the GEANT collector. There are many technical deployments of federation Fticks agregator, starting from simple syslog server, to a more robust and complex log management system, such as ELK stack.

Following are guidelines on how to deploy ELK stack as federation level Fticks aggregator.

  • Deploy Logstash as a Fticks aggregator
  • Configure Logstash to listen to port 514
    • Logstash can be configured to listen to ephemeral ports (eg. 1514/udp)

      input {
        syslog {
          port => 1514
          codec => plain
          syslog_field => "syslog"
        }
      }
      

    • Configure persistent iptables rule to forward all udp traffic from port 514/udp to 1514/udp

      iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to-port 1514
      

    • Configure Logstash to process the received Ftick (for federation processing purposes)

       
      filter{
        mutate{
          remove_tag => [ "_grokparsefailure_sysloginput" ]
        }
      
        grok{
          match => [ "message", "(<%{POSINT:priority1}>%{SYSLOGTIMESTAMP:timestamp1} %{GREEDYDATA:hostname1})?<%{POSINT:priority}>%{SYSLOGTIMESTAMP:timestamp} %{GREEDYDATA:hostname} (%{GREEDYDATA:appname} )?\[%{GREEDYDATA:process}\](\:)? %{GREEDYDATA:syslog_message}" ]
        }
        if "_grokparsefailure" in [tags] {
          mutate{
            remove_tag => [ "_grokparsefailure" ]
            add_tag => [ "not_ftick" ]
          }
        }
      
        grok{
          match => [ "syslog_message", "(%{GREEDYDATA:bean} )?F-TICKS/%{GREEDYDATA:federation}/%{GREEDYDATA:version}[#]TS=%{GREEDYDATA:TS}[#]RP=%{GREEDYDATA:RP}[#]AP=%{GREEDYDATA:AP}[#]PN=%{GREEDYDATA:PN}([#]CSI=%{GREEDYDATA:CSI})?([#]AM=%{GREEDYDATA:AM})?([#]RESULT=%{GREEDYDATA:RESULT})?[#]" ]
        }
        if "_grokparsefailure" in [tags] {
          mutate{
            remove_tag => [ "_grokparsefailure" ]
          }
          grok{
            match => [ "syslog_message", "(%{GREEDYDATA:bean} )?F-TICKS/%{GREEDYDATA:federation}/%{GREEDYDATA:version}[#]RESULT=%{GREEDYDATA:RESULT}([#]CSI=%{GREEDYDATA:CSI})?[#]AP=%{GREEDYDATA:AP}[#]RP=%{GREEDYDATA:RP}[#]PN=%{GREEDYDATA:PN}[#]TS=%{GREEDYDATA:TS}[#]" ]
          }
        }
      
        grok {
           match => [ "RP", "(%{URIPROTO:RP_uri_proto}://(?:%{USER:RP_user}(?::[^@]*)?@)?)?(?:%{URIHOST:RP_uri_domain})?(?:%{URIPATHPARAM:RP_uri_param})?" ]
        }
      
        grok {
           match => [ "AP", "(%{URIPROTO:AP_uri_proto}://(?:%{USER:AP_user}(?::[^@]*)?@)?)?(?:%{URIHOST:AP_uri_domain})?(?:%{URIPATHPARAM:AP_uri_param})?" ]
        }
      
        geoip {
           source => "AP_uri_domain"
           target => "AP_geoip"
        }
      
        geoip {
           source => "RP_uri_domain"
           target => "RP_geoip"
        }
      }
      

    • Configure Logstash to forward all Fticks to the central GENAT aggregator

      output {
      
          # output to central GEANT Fticks aggregator
          syslog {
            host => "90.147.166.156"
            message => "%{syslog_message}"
            codec => "plain"
            port => 514
          }
      
          # output to a local elasticsearch index for federation level visualizations
          elasticsearch {
            id => "fticks"
            hosts => [ "localhost:9200" ]
            index => "fticks_edugain"
          }
      
      }
      

    • For better indexing of data, we need to put an index template in the elasticsearch configuration

      $ wget https://raw.githubusercontent.com/GEANT/FTicks-ELK/master/haproxy/config/fticks.template
      $ wget https://raw.githubusercontent.com/GEANT/FTicks-ELK/master/haproxy/config/fticks.index
      $ curl -X PUT -H 'Content-Type: application/json' http://localhost:9200/_template/fticks_template -d '@fticks.template'
      $ curl -X PUT -H 'Content-Type: application/json' http://localhost:9200/fticks -d '@/fticks.index'