Wiimms mkw-ana (tool)

Wiimms mkw-ana

File:Wiimms-SZS-Tools.png
Author:	Wiimm
Operating Systems:	Linux (i386,x86_64), Windows (Cygwin).
Software Type:	Mario Kart Wii network traffic analyzer
File Formats:	PCAP 2.4 (tcpdump), bzip2 compression, BMG (text).
Current Version:	0.13, 2013-09-07

The tools mkw-ana is a new project by Wiimm to analyze the network protocol of Mario Kart Wii.

Intention

In November 2012, Wiimm decided to analyze the network protocol of Mario Kart Wii. The main goal was to discover online cheaters. Another goal is to set up his own server if any time Nintendo will shut down its servers.

So the main feature is to dump the packets of a tcpdump (done by tcpdump or wireshark) in a user-friendly format. After first experiments, it becomes also a live racing statistic tool.

The Tool

Change log

mkw-ana v0.13 r1683 - 2013-09-07

 - Improved built-in help system. Type "mkw-ana help help" for details.

 - New option --ana=file: Opens a output file to store analysis data.

 - New option: --ana-mode=list (short: --amd=list):
   Print only the specified events to the analysis file.

 - New options --hms and --hms-info to print relative timestamps in HH:MM:SS.

Old change log

Show text

mkw-ana v0.10 r1577 - 2013-06-01

 - Complete new RACE command: Print a live race table of all players with the
   following infos:
    - Rank in race
    - Position on map
    - Lakitu status
    - Drift status
    - Drift counters
    - Friend code
    - Mii name
 - First cheat check: The new option --drift enables the checking of drift
   stages (drift,blue,yellow). For command LOG and RACE it is always enabled.
   *** This cheat check is EXPERIMENTAL! ***
 - Option --delta works now nibble based instead of byte based.
 - Several small improvements.


mkw-ana v0.11 r1622 - 2013-07-14

 - More record types are detected now, some only as unnamed hex-records.
 - The 'PREFIX*' records are renamed to 'ROOM*', because they are only send in
   room events. For non room events, there exists the prefix record 'GLOBAL'.
 - Optimized event detection, including world wide races and battles.
 - A first battle stage detection works now. Battle records are different from
   racing rcords and not analysed yet.
 - New command: COUNT: Count the record types. Together with option --verbose
   it shows all defined record types with short and long names.
 - Event and race numbers are now incremented at begin of an event.
 - Track, driver, vehicle and engine names are now available in (us)english
   and german without loading a BMG text file. The default output language is
   english. If option --de is set, the default output language is switched to
   german. However, a loaded BMG text file may override all names.
 - Bug fix: Daylight saving is now recognized.
 - The checksum of each UDP packet is now checked and invalid packets are
   dropped silently. If option --checksum is set, the packets are not dropped.
   This makes only sense for hex dumps. If option --checksum is set twice, the
   checksums are not calculated and assumed to be correct.
 - Some minor printing and counting bugs fixed.
 - Windows only: Cygwin update to v1.7.20 2013-06-07.
 - Windows only: The needed Cygwin DLL files are now delivered too.

mkw-ana v0.12 r1644 - 2013-08-19

 - Bug fix: For aligment, the Mii data in user records is 0x4c bytes long,
   2 bytes more than the Mii file size (0x4a=74).
 - Detection of new records: ITEM_CTRL, ITEM_LIST, ITEM_DATA.
 - First item cheat detection, only visible for command RACE. It produce
   false positives, but if the counter is >5, a cheater play with you.
 - Command RACE: New output with head line, speed and item cheat infos.
 - New option for command RACE: --print=list (-P list):
    Print only the specified columns of the output table. A comma separated
    list of keywords is expected: RANK, TIME, POS=X-POS+Y-POS+Z-POS, DIR,
    SPEED=3D-SPEED,H-SPEED, STATUS, DRIFT=D-COUNT+D-CHEAT,
    XDRIFT=DRIFT+D-MINTIME, ITEM=I-CHEAT, CHEAT=D-CHEAT,I-CHEAT TEST, FC,
    MINI-FC and NAME. Also available: CLEAR, MIN, DEFAULT, MAX and ALL.

Built-in Help

Let's start with the built-in help as an overview about the tool:

Show text

mkw-ana v0.13/x86_64 r1683 -- Dirk Clemens -- 2013-09-07
--------------------------------------------------------

mkw-ana : Analyze network dumps (created by tcpdump) and print summaries.

Syntax: mkw-ana [option]... command [option|parameter|file]...


Commands:

  VERSION         : Print program name and version and exit.
  HELP      | H   : Print help for commands and options.
  ARGTEST         : This debug command accepts all kinds of parameters and
                    prints one line for each parameter or option.
  TEST            : Test options: All options are allowed, some are printed.
  ERROR     | ERR : Translate exit codes to message names. If no exit code is
                    entered, print a table with all error messages.

  DUMP0     | D0  : Print a raw dump of all packets. This dump can be used for
                    all network dumps, not only for MKWii.
  DUMP1     | D1  : Packet based hex dumper.
  DUMP2     | D2  : First record based hex dumper.
  DUMP3     | D3  : New and improved variant of DUMP2 (record based).
  DUMP      | D   : Use the best/latest dumping mode (depends on options). At
                    the moment DUMP is an alias for DUMP3.

  FLOWRATES | F   : Print flowrates of the data traffic. This command can be
                    used for all network dumps, not only for MKWii.
  DNS             : Print DNS and optional ARP packets in human readable
                    format. This command can be used for all network dumps, not
                    only for MKWii.

  SILENT    | SIL : Iterate and analyse the source files, but print nothing.
  CLIENTS   | C   : List all clients.
  USERS     | U   : List all users.
  RECORDS   | R   : Print all record names.
  COUNT     | CNT : Count the record types.
  STAGES    | S   : Print all stages.
  LOG       | L   : Print all stages including tables.
  TRACKS          : Print all track selections.
  TOTALS    | T   : Print all totals as text dump.

  RACE            : Live race table of all players. The screen is updated every
                    0.5 seconds.

Type 'mkw-ana HELP command' to get command specific help.

Global options:

  -V --version       Stop parsing the command line, print a version info and
                     exit.
  -h --help          Print help and exit. If the first non option is a valid
                     command name, then a help for the given command is
                     printed.
     --xhelp         Stop parsing the command line and print a help message
                     with all commands included. Exit after printing.
     --width width   Define the width (number of columns) for help and some
                     other messages and disable the automatic detection of the
                     terminal width.
  -q --quiet         Be quiet and print only error messages. All previous
                     --verbose are canceled. Multiple usage is possible. The
                     impact is command dependent.
  -v --verbose       Be verbose and print more progress information. All
                     previous --quiet are canceled. Multiple usage is possible.
                     The impact is command dependent.
  -A --allow-all     Usually commands accept only options with impact to the
                     command. All other options fire a syntax error. But if
                     --allow-all is set, all commands accept all options.
                       This makes changing the command of a long command line
                     without removing useless options easier. It also helps to
                     override wrong option permissions.
     --de            Use german names.
     --old           Use old implementation if available. All previous --new
                     are canceled.
     --new           Use new implementation if available. All previous --old
                     are canceled.

     --bmg file      Read a BMG text file to scan online chat messages and
                     track, driver and vehicle names. Disable auto load of BMG
                     files. Multiple usage is possible.
     --team file     Read a text file for team assignments and disable auto
                     load of team files. Multiple usage is possible.
     --origin x,y,z  Define an alternative origin for positions.
     --rel           Print timestamps as seconds relative to the beginning.
                     Dependent of option --long the formats are: 'SSSSS',
                     'SSSSS.s' or 'SSSSS.sss'
     --rel-info      Like --rel, but reset the origin whenever a reference time
                     is defined in the info file.
     --hms           Enable relative time stamps Like --rel, but print them in
                     HH:MM:SS instead in seconds only.
     --hms-info      Short cut for '--rel-info --hms'.
     --ana file      Open a log file and dump text lines for further analysis.
                     The first word of each line classified the output type. If
                     first character of 'file' is a '+', append data to an
                     already existent file. If the filename is only '-', then
                     dump to stdout.
     --ana-mode list Print only the specified events to the analysis file.
                     --amd is a short cut. A comma separated list of keywords
                     is expected: CHEATS=IT-CHEATS, XCHEATS=CHEATS,IT-XCHEATS,
                     ITEM, EV-DLEN, EV-ALL-DLEN, EV-NAME. Also available:
                     CLEAR, DEFAULT and ALL. If flag SINGLE is set, repeat
                     count support is disabled. If flag FLUSH is set, the
                     output is flushed for each line.

Command specific options with common description:


     --adjust float  Adjust time stamps of the network dump by adding 'float'
                     seconds. This may help to synchronize different dumps.
     --skip float    Skip first 'float' seconds of each read network dump.
                     Negative values are relative to the end (or ignored for
                     pipes).
     --term float    Terminate each dump at 'float' seconds. Negative values
                     are relative to the end (or ignored for pipes).
     --combine       Logical combine network dumps to one single dump before
                     executing options --skip and --term.
     --checksum      Normally, UDP packets with wrong checksums are dropped. If
                     --checksum is set, the checksums are calculated, but no
                     packet is dropped. Some dumps will print a status info. If
                     set twice, checksums are never calculated and assumed to
                     be correct. --csum is a short cut.
  -f --follow        Don't close the last input dump on reaching end of file.
                     Instead wait for appended data. This works like the unix
                     tool 'tail -f'.
     --ip addr       Define an address (IP or DNS name) for filtering. Only
                     packets from and to this host are accepted, all others are
                     ignored.
     --home addr     Define an address (IP or DNS name) as home client.
                       Without this options, the tool tries to determine the
                     home client by analysing sender and receiver of the first
                     non filtered packet. A local network (10/8, 172.16/12,
                     192.168/16, 169.254/16) has priority over a non local
                     network. If sender and receiver have the same priority,
                     the IP of the sender is used.
     --wii addr      Define an address (IP or DNS name) as home client and for
                     filtering. This options is a shortcut for '--home addr
                     --ip addr'.
     --write file    Write filtered network packets as PCAP v2.4 to 'file'.

  -p --no-proxy      Don't dump proxy packets (packets, which contains a PROXY
                     record).
  -r --receive       Dump only network packets received by the home client
                     (option --home) or allowed by --send, --receive-ip or
                     --send-ip.
  -s --send          Dump only network packets send by the home client (option
                     --home) or allowed by --receive, --send-ip or
                     --receive-ip.
     --receive-ip addr 
                     Dump only network packets received by the entered address
                     (IP or DNS name) or allowed by --send-ip, --receive or
                     --send. --rip is a short cut for --receive-ip.
     --send-ip addr  Dump only network packets send by the entered address (IP
                     or DNS name) or allowed by --receive-ip, --send or
                     --receive. --sip is a short cut for --send-ip.
     --transfer-ip addr 
                     Dump only network packets receiced or send by the entered
                     address (IP or DNS name) or allowed by --send or
                     --receive. --tip is a short cut for --transfer-ip and both
                     are short cuts for '--rip addr --sip addr.
  -L --length ranges Dump only UDP packets with specified UDP data length. The
                     8 bytes long UDP header does not count.
                       The parameter is a comma separated list of INDEX,
                     INDEX1:, INDEX1:INDEX2 and INDEX#LENGTH elements.
  -S --stage list    Dump UDP packets only, if one of the entered stages is
                     active.
                       The parameter is a comma separated list of stage names,
                     optional preceeded by '+' (enable) or '.' (disable). Type
                     'mkw-ana test' for a list of stages or use the dumps to
                     identify stage names.
     --xevent        Support the XEVENT record type. It is an overlay over the
                     ITEM and EVENT records. --xeve is a shortcut. The option
                     is automatically set, if --type or --TYPE call the XEVENT
                     record.
  -t --type list     Dump UDP packets only, if at least one record of the
                     packet match the entered record list.
                       The parameter is a comma separated list of record names,
                     optional preceeded by '+' (enable) or '.' (disable). Type
                     'mkw-ana test' for a list of records or use the dumps to
                     identify record names.
  -T --TYPE list     Same as --type except for command DUMP3.

  -b --brief         If set once, the dump header (timestamp and client info)
                     of single line dumps becomes smaller. If set twice,
                     timestamp and client info are not printed. All previous
                     --long are canceled.
  -l --long          This option is relevant for single line dumps. Usually the
                     time format is printed as 'MM:SS.s' to keep the lines
                     small. If set once, 'HH:MM:SS.s' is used. If set twice,
                     'HH:MM:SS.sss' is used. All previous --brief are canceled.
     --list          Print a list of events instead of a summary.
  -1 --one-line      Print the hexdumps as one line for each record. This makes
                     the dumps horizontal very large, but it is good for
                     comparing objects of the same type. Very helpful is to
                     pipe the output to 'less -S', which supports horizontal
                     scrolling.
                       If set twice, some record types are not split into sub
                     records.
  -2 --sep-lines     Dump one line per record (like option --one-line) and an
                     empty line between packets.
  -a --ascii         Append an ASCII character dump behind the hexdump. This
                     option is ignored if using --one-line or --sep-lines.
  -n --native        If set, some known values are printed in native format
                     instead as simple hex number. If set twice, some other
                     values, that will destroy the column layout of the
                     hexdump, will printed in native format too.
  -d --delta         If set, record data is compared with the data of the
                     previous record of same type and client. If a nibble (4
                     bits) is unchanged, a '-' is printed intead of a hex
                     digit.
  -I --index ranges  Dump only bytes with an index selected by the range list.
                     This make the hex dump smaller especially for one-line
                     dumps.
                       The parameter is a comma separated list of INDEX,
                     INDEX1:, INDEX1:INDEX2 and INDEX#LENGTH elements.
  -P --print list    Print only the specified columns of the output table. A
                     comma separated list of keywords is expected: RANK, TIME,
                     POS=X-POS+Y-POS+Z-POS, DIR, SPEED=3D-SPEED,H-SPEED,
                     STATUS, DRIFT=D-COUNT+D-CHEAT, XDRIFT=DRIFT+D-MINTIME,
                     ITEM=I-CHEAT+I-COUNT+I-SUMMARY, CHEAT=D-CHEAT,I-CHEAT, FC,
                     WHO=MINI-FC+NAME. Also available: NONE, MIN, DEFAULT, MAX,
                     ALL.

     --min-race num  This is a statistic option: If a Grand Prix (single or
                     team) is aborted, the results of the Grand Prix are only
                     used in the statstics, if NUM races has been completed.
                     The default is 2 and possible values are 0..4.
     --drift         Print drift statistics during logging.

     --log file      Log into the file using the same output as command LOG. If
                     first character of 'file' is a '+', append data to an
                     already existent file. If the filename is only '-', then
                     log to stdout.
     --md file       Create a MakeDoc script with results after each race.
     --mdx file      Create a MakeDoc script with results after each race. Same
                     as --md, but replace %E, %R, %N and %T in the filename by
                     'event id', 'race id', 'total race' and 'event type' to
                     create different files.
     --php file      Create a php script with results after each race.
     --phpx file     Create a php script with results after each race. Same as
                     --php, but replace %E, %R, %N and %T in the filename by
                     'event id', 'race id', 'total race' and 'event type' to
                     create different files.
     --sleep float   Sleep 'float' seconds after a race has finished. This
                     option slows down a simulation run direct after logging
                     and printing the race statistics. Start with values of the
                     range from 3 to 15 seconds.

     --mii dir       Extract Miis to the already existing directory 'dir'.
                     Existing Mii files will be overwritten.

General Description

The tool started as simple hex dumper reading network dumps in PCAP format. In the first phase of the tool, the textual dumps of wireshark and tcpdump were much better. But after only a few days, the tool learned to handle records, clients, users, friend codes and Miis. From this moment the tool was better to analyze the Mario Kart Wii traffic.

Now, mkw-ana split the traffic into records and scans some data to detect stages of the online meeting. Stages are for example room, prepare race, count down, racing and end of race. It is able to separate races into events (grand prix and team rand prix) and to calculate racing tables. Racing data can also be exported to support live statistics.

At the moment there are three different kinds of hexdumps. All 3 are able to dump in one line mode to have large tables. Tool less is here a very good tool for vertical and horizontal scrolling. The stages are includes into the dump as comment lines. The dumped records can be filters by sending, receiving, proxy, record types, stage types and packet length. It is also possible to select the dumped bytes by indices and ranges.

Another feature is, that mkw-ana can read comment files. If making videos of the dumped meetings, you can write such comment file. Each line starts with a timestamp followed by a comment. Virtual Dub is a good tool for this job. Then you must synchronize the comment file with the network dump. The start of the first game ("GO" in the video) is a very good point for synchronization. Here is an example of a comment file (in german):

>2012-12-05 19:10:39.745 - 12:22.792
0:00:00.000 Video Start

0:12:22.792 Rennen 1.1, GO!
0:12:54.123 Tinti wird angekündigt, T=0:31.322
0:14:17.924 Power wird angekündigt, T=1:55.120
0:15:33.633 Blitz schlägt ein, T=1:35.326
...

Notes

• The first line is the real time of the start of the race minus the video time stamp. This is the synchronisation. An synchronisation can be done multiple times.
• The line with video timestamp and comment follow. The focus of the comment change as the point of interest.
• The name of the comment file must be the same as the network dump, but it must have the extension ".info" instead of ".eth".

???

Download

You can find the latest and some old distributions here

• http://download.wiimm.de/mkw-ana/

Content

• Binaries for:
  • Linux i386
  • Linux x86_64
  • Cygwin/Windows (Needed Cygwin DLL files are delivered. Best is to install a Cygwin system).
• Some scripts as examples.
• Some BMG text examples.
• Some doc files.

Sometimes I upload single tool updates for testers

• http://download.wiimm.de/mkw-ana/bin/

And remember

It is an alpha distribution and and only minimal support will be given at the moment.

Capture the network data

First you must capture the network traffic of the Wii. Therefore you must redirect it to a PC running a capture software. There are 3 general ways to to this:

• If you have a manageable switch, enable port mirroring and send all Wii traffic to a PC.
• Use your PC as router.
• Use old network hubs (not switches). A hub will mirror all traffic of all ports to all others; it's just a multi port repeater and will slow down your network.

Use a software like tcpdump or wireshark to capture the data. Best is to save the captured data directly to a file or to send it to other commands like mkw-ana for a live analysis.

It's also possible to save the data to a file and to make a live analysis at the same time. Use the following command pipe:

tcpdump -w- -U -i eth1 host wii | tee save.dump | mkw-ana ...

It is important to filter the data by host ip_or_name, because foreign traffic interfere the wii traffic analysis and will have curious side effects.

If using wireshark, save the dump to a file and use the following command for a live analysis:

mkw-ana --follow DUMPFILE ...

Live Statistics

mkw-ana can scan the network traffic in real time and can produce makedoc or php data files. Together with ssh and an cgi script, a live statistic is created. Live means that the tables are updates 2-5 seconds after the race have finished.

How it works

The whole job is done by 3 processes:

1. First, you must capture the network traffic like described above. Then use one of the commands:

   ... | mkw-ana log --md  DATAFILE
   ... | mkw-ana log --php DATAFILE
   mkw-ana --follow DUMPFILE log --md  DATAFILE
   mkw-ana --follow DUMPFILE log --php DATAFILE
   

2. Each time, a new DATAFILE is written, it must be transferred to the web server. A script using shh, sftp scp or ftp within an endless loop will do this job automatically.
3. Last not least, a CGI or PHP script running at the web server must read the data files to serve a html-page to the visitors.

To see, what live means, visit the live statistics on Wednesday or Thursday between 19:10 and 20:30 CET (Central European Time).

Links

• MKWii Network Protocol
• Discussion at mariokartwii.com

References

Template:MKWii Network Protocol