Difference between revisions of "Wiimms mkw-ana (tool)"

From Custom Mario Kart
Jump to navigation Jump to search
(Wikipedia instead of MediaWiki)
(mkw-ana v0.18)
Line 16: Line 16:
 
|-
 
|-
 
! Current Version:
 
! Current Version:
| v0.17, 2014-03-03
+
| v0.18, 2014-03-24
 
|}
 
|}
  
Line 40: Line 40:
  
 
<pre>
 
<pre>
 +
mkw-ana v0.18 r1836 - 2014-03-24
 +
 +
- New command: FRIENDCODES: A friend code calculator. For each scanned
 +
  parameter a line with a decimal friend code, a hexadecimal friend code,
 +
  a decimal player ID and a game ID is printed. As input, various formats
 +
  are supported. The new option --gid defines a game ID for creating a FC
 +
  from a player id.
 +
 +
- New command: ANALYZE: Analyze the dump files for special information and
 +
  print it a machine readable list, one line per record. The first paraneter
 +
  a comma separated list of keywords: PARAM-NAMES, NICKS, ALL.
 +
  See built-in help for details.
 +
 +
- New letter ID scheme for clients. All IDs are used circulary:
 +
    0-9: Known Nintendo servers (detected by a previous DNS query).
 +
    A-Z: Home client (--ip or --wii or atuo detect).
 +
    a-z: Other clients (players).
 +
 +
- Internal statistics about UDP (old) and TCP (new) traffic (send+receive
 +
  packtes+bytes). The client dumps print now a summary of TCP+UDP.
 +
 +
- Command 'CLIENT' prints the clients in order of appearance. If option
 +
  --long is set, a multi line list with more details is printed.
 +
 +
- The new command SERVERS prints a multi line list like 'CLIENTS --long',
 +
  but only Nintendos servers are selected.
 +
 +
- Command COUNT will now count also ARP, DNS and TCP records.
 +
 +
- All commands can be prefixed by a 'C' (e.g. 'CDUMP' instead of 'DUMP') to
 +
  force colorized output (= force option --color).
 +
 +
- Options --ip and --wii accept now an optional port number.
 +
 +
- Bug fix for 'A' detection of DNS replies.
 +
 +
 
mkw-ana v0.17 r1786 - 2014-03-03
 
mkw-ana v0.17 r1786 - 2014-03-03
  
Line 51: Line 88:
  
 
  - Some minor optical and layout changes.
 
  - Some minor optical and layout changes.
 +
</pre>
  
 
+
;Old change log:
 +
<spoiler><pre>
 
mkw-ana v0.16 r1765 - 2014-01-18
 
mkw-ana v0.16 r1765 - 2014-01-18
  
Line 61: Line 100:
 
       supported to retrieve packets.  
 
       supported to retrieve packets.  
 
     - Optional bzip2 compression of all dump formats.
 
     - Optional bzip2 compression of all dump formats.
 
 
  - Complete new handling of ARP, DNS and TCP packets. They will now be handled
 
  - Complete new handling of ARP, DNS and TCP packets. They will now be handled
 
   as records and are written to the output file, if option --write is set.
 
   as records and are written to the output file, if option --write is set.
 
 
  - New record types: ARP, DNS, TCP, QUERY, TPARAM, UPARAM.
 
  - New record types: ARP, DNS, TCP, QUERY, TPARAM, UPARAM.
 
 
  - New global option: --wide[=width]: Usually hexdumps cover 16 bytes per
 
  - New global option: --wide[=width]: Usually hexdumps cover 16 bytes per
 
   line. If option --wide is set, 32 bytes per line are covered. Optional it
 
   line. If option --wide is set, 32 bytes per line are covered. Optional it
 
   is possible to enter a value.
 
   is possible to enter a value.
 
 
  - New keywords for option --log-mode:
 
  - New keywords for option --log-mode:
 
     'QUERY'  : Dump database quers (DB,table,select,where).
 
     'QUERY'  : Dump database quers (DB,table,select,where).
 
     'REGION' : Dump region and world wide queries.
 
     'REGION' : Dump region and world wide queries.
 
     'TCP'    : Shortcut for QUERY,REGION: Log all TCP related stuff.
 
     'TCP'    : Shortcut for QUERY,REGION: Log all TCP related stuff.
 
 
  - New options: Option --color forces colorized text (where supported).
 
  - New options: Option --color forces colorized text (where supported).
 
   It is enabled by default for output to terminals. Option --no-color
 
   It is enabled by default for output to terminals. Option --no-color
 
   disables colorized text at all.
 
   disables colorized text at all.
 
 
  - New command: COLORS: Test colorized text by printing it in different modes
 
  - New command: COLORS: Test colorized text by printing it in different modes
 
   (colors, bold, underline). Also test the options --color and --no-color.
 
   (colors, bold, underline). Also test the options --color and --no-color.
 
 
  - New command LIST: List all stage or record names.
 
  - New command LIST: List all stage or record names.
 
 
  - More options to filter packets for dumps: --and, --receive-mac (--rmac),
 
  - More options to filter packets for dumps: --and, --receive-mac (--rmac),
 
   --send-mac (--smac) and --transfer-mac (--tmac).
 
   --send-mac (--smac) and --transfer-mac (--tmac).
 
 
  - New options to control MAC printing in dumps: --show-mac and --hide-mac.
 
  - New options to control MAC printing in dumps: --show-mac and --hide-mac.
 
 
  - The new options --real-time-factor (--rtf) and --real-time-wait (--rtw)
 
  - The new options --real-time-factor (--rtf) and --real-time-wait (--rtw)
 
   help to analyze old dumps in real time, time-laps or slow-motion.
 
   help to analyze old dumps in real time, time-laps or slow-motion.
  
 
</pre>
 
 
;Old change log:
 
<spoiler><pre>
 
 
mkw-ana v0.15 r1703 - 2014-01-04
 
mkw-ana v0.15 r1703 - 2014-01-04
  
Line 105: Line 129:
 
   are included into the log file or output. Allowed keywords are: STATUS,
 
   are included into the log file or output. Allowed keywords are: STATUS,
 
   SELECT, DRIVER, RACE, EVENT, TOTAL, CHEATS, NONE and ALL (default).
 
   SELECT, DRIVER, RACE, EVENT, TOTAL, CHEATS, NONE and ALL (default).
 
  
 
mkw-ana v0.14 r1688 - 2013-10-19
 
mkw-ana v0.14 r1688 - 2013-10-19
Line 112: Line 135:
 
   than 32 tracks). Allowed keywords are DISABLED, AUTO (default), ENABLED.
 
   than 32 tracks). Allowed keywords are DISABLED, AUTO (default), ENABLED.
 
   Without parameter, CT-CODE support is enabled.
 
   Without parameter, CT-CODE support is enabled.
 
  
 
mkw-ana v0.13 r1683 - 2013-09-07
 
mkw-ana v0.13 r1683 - 2013-09-07
Line 121: Line 143:
 
   Print only the specified events to the analysis file.
 
   Print only the specified events to the analysis file.
 
  - New options --hms and --hms-info to print relative timestamps in HH:MM:SS.
 
  - New options --hms and --hms-info to print relative timestamps in HH:MM:SS.
 
 
mkw-ana v0.12 r1644 - 2013-08-19
 
 
- Bug fix: For aligment, the Mii data in user records is 0x4c bytes long,
 
  2 bytes more than the Mii file size (0x4a=74).
 
- Detection of new records: ITEM_CTRL, ITEM_LIST, ITEM_DATA.
 
- First item cheat detection, only visible for command RACE. It produce
 
  false positives, but if the counter is >5, a cheater play with you.
 
- Command RACE: New output with head line, speed and item cheat infos.
 
- New option for command RACE: --print=list (-P list):
 
    Print only the specified columns of the output table. A comma separated
 
    list of keywords is expected: RANK, TIME, POS=X-POS+Y-POS+Z-POS, DIR,
 
    SPEED=3D-SPEED,H-SPEED, STATUS, DRIFT=D-COUNT+D-CHEAT,
 
    XDRIFT=DRIFT+D-MINTIME, ITEM=I-CHEAT, CHEAT=D-CHEAT,I-CHEAT TEST, FC,
 
    MINI-FC and NAME. Also available: CLEAR, MIN, DEFAULT, MAX and ALL.
 
 
</pre></spoiler>
 
</pre></spoiler>
  
Line 145: Line 151:
 
<spoiler><pre>
 
<spoiler><pre>
  
mkw-ana v0.17/x86_64 r1786 -- Dirk Clemens -- 2014-03-03
+
mkw-ana v0.18/x86_64 r1836 -- Dirk Clemens -- 2014-03-24
 
--------------------------------------------------------
 
--------------------------------------------------------
  
Line 155: Line 161:
 
Commands:
 
Commands:
  
   VERSION         : Print program name and version and exit.
+
   VERSION           : Print program name and version and exit.
   HELP     | H  : Print help for commands and options.
+
   HELP       | H  : Print help for commands and options.
   ARGTEST         : This debug command accepts all kinds of parameters and
+
   ARGTEST           : This debug command accepts all kinds of parameters and
                    prints one line for each parameter or option.
+
                      prints one line for each parameter or option.
   TEST           : Test options: All options are allowed, some are printed.
+
   TEST             : Test options: All options are allowed, some are printed.
   COLORS         : Ignore all parameters and print clored text as test.
+
   COLORS           : Ignore all parameters and print clored text as test.
   ERROR     | ERR : Translate exit codes to message names. If no exit code is
+
   ERROR       | ERR : Translate exit codes to message names. If no exit code is
                    entered, print a table with all error messages.
+
                      entered, print a table with all error messages.
   LIST           : List the keywords of a class. Allowed classes are: STAGES,
+
   LIST             : List the keywords of a class. Allowed classes are:
                    RECORDS and ALL.
+
                      STAGES, RECORDS and ALL.
 +
  FRIENDCODES | FC  : Calculate friend codes and player ids. For each scanned
 +
                      parameter a line with a decimal friend code, a
 +
                      hexadecimal friend code, a decimal player ID and a game
 +
                      ID is printed. Various formats are accepted for input.
  
   DUMP0     | D0  : Print a raw dump of all packets. This dump can be used for
+
   DUMP0       | D0  : Print a raw dump of all packets. This dump can be used
                    all network dumps, not only for MKWii.
+
                      for all network dumps, not only for MKWii.
   DUMP1     | D1  : Packet based hex dumper.
+
   DUMP1       | D1  : Packet based hex dumper.
   DUMP2     | D2  : First record based hex dumper.
+
   DUMP2       | D2  : First record based hex dumper.
   DUMP3     | D3  : New and improved variant of DUMP2 (record based).
+
   DUMP3       | D3  : New and improved variant of DUMP2 (record based).
   DUMP     | D  : Use the best/latest dumping mode (depends on options). At
+
   DUMP       | D  : Use the best/latest dumping mode (depends on options). At
                    the moment DUMP is an alias for DUMP3.
+
                      the moment DUMP is an alias for DUMP3.
  
   FLOWRATES | F  : Print flowrates of the data traffic. This command can be
+
   FLOWRATES   | F  : Print flowrates of the data traffic. This command can be
                    used for all network dumps, not only for MKWii.
+
                      used for all network dumps, not only for MKWii.
   DNS             : Print DNS and optional ARP packets in human readable
+
   DNS               : Print DNS and optional ARP packets in human readable
                    format. This command can be used for all network dumps, not
+
                      format. This command can be used for all network dumps,
                    only for MKWii.
+
                      not only for MKWii.
  
   SILENT   | SIL : Iterate and analyse the source files, but print nothing.
+
   SILENT     | SIL : Iterate and analyse the source files, but print nothing.
   CLIENTS   | C   : List all clients.
+
   ANALYZE    | ANA : Analyze the dump files for special information and print
   USERS     | U  : List all users.
+
                      it a machine readable list, one line per record.
   RECORDS   | R  : Print all record names.
+
                      Parameter MODELIST is a comma separated list of keywords.
   COUNT     | CNT : Count the record types.
+
                      Each keyword enables one kind of analysis:
   STAGES   | S  : Print all stages.
+
                      * PARAM-NAMES: Print names of STRING-PARAM records, one
   LOG       | L  : Print all stages including tables.
+
                      line for each LIST.
   TRACKS         : Print all track selections.
+
                      * NICKS: Collect data about user and their nicks and
   TOTALS   | T  : Print all totals as text dump.
+
                      friend lists.
 +
                      * ALL: All of above.
 +
   SERVERS    | SER : Print statistics about all servers.
 +
   CLIENTS    | CLI : Print statistics about all clients.
 +
   USERS       | U  : Print statistics about all users.
 +
   RECORDS     | R  : Print all record names.
 +
   COUNT       | CNT : Count the record types.
 +
   STAGES     | S  : Print all stages.
 +
   LOG         | L  : Print all stages including tables.
 +
   TRACKS           : Print all track selections.
 +
   TOTALS     | T  : Print all totals as text dump.
  
   RACE           : Live race table of all players. The screen is updated every
+
   RACE             : Live race table of all players. The screen is updated
                    0.5 seconds.
+
                      every 0.5 seconds.
  
 
Type 'mkw-ana HELP command' to get command specific help.
 
Type 'mkw-ana HELP command' to get command specific help.
Line 286: Line 306:
 
                     Instead wait for appended data. This works like the unix
 
                     Instead wait for appended data. This works like the unix
 
                     tool 'tail -f'.
 
                     tool 'tail -f'.
     --ip addr       Define an address (IP or DNS name) for filtering. Only
+
     --ip addr[:port]
                    packets from or to this host are accepted, all others are
+
                    Define an address (IP or DNS name) and optional a port for
                    ignored.
+
                    filtering. Only packets from or to this host are accepted,
 +
                    all others are ignored.
 
     --home addr    Define an address (IP or DNS name) as home client.
 
     --home addr    Define an address (IP or DNS name) as home client.
 
                       Without this options, the tool tries to determine the
 
                       Without this options, the tool tries to determine the
Line 296: Line 317:
 
                     network. If sender and receiver have the same priority,
 
                     network. If sender and receiver have the same priority,
 
                     the IP of the sender is used.
 
                     the IP of the sender is used.
     --wii addr     Define an address (IP or DNS name) as home client and for
+
     --wii addr[:port]
                    filtering. This options is a shortcut for '--home addr
+
                    Define an address (IP or DNS name) and optional a port as
                    --ip addr'.
+
                    home client and for filtering. This options is a shortcut
 +
                    for '--home addr --ip addr:port'.
 
     --write file    Write filtered network packets as PCAP v2.4 to 'file' with
 
     --write file    Write filtered network packets as PCAP v2.4 to 'file' with
 
                     local endian and microseconds format.
 
                     local endian and microseconds format.
Line 455: Line 477:
 
     --mii dir      Extract Miis to the already existing directory 'dir'.
 
     --mii dir      Extract Miis to the already existing directory 'dir'.
 
                     Existing Mii files will be overwritten.
 
                     Existing Mii files will be overwritten.
 +
    --gid game_id  Define the game ID for friend code calculations. Up to 4
 +
                    characters of the parameter replace the default value RMCJ
 +
                    (Mario Kart Wii).
  
 
</pre></spoiler>
 
</pre></spoiler>

Revision as of 21:03, 24 March 2014

Wiimms mkw-ana
File:Wiimms-SZS-Tools.png
Author: Wiimm
Operating Systems: Linux (i386,x86_64),
Windows (Cygwin).
Software Type: Mario Kart Wii network traffic analyzer
File Formats: PCAP 2.4, PCAP 2.4.modified, PCAP-NG, bzip2 compression, BMG (text).
Current Version: v0.18, 2014-03-24

The tools mkw-ana is a new project by Wiimm to analyze the network protocol of Mario Kart Wii.



Intention

In November 2012, Wiimm decided to analyze the network protocol of Mario Kart Wii. The main goal was to discover online cheaters. Another goal is to set up his own server when Nintendo shuts down its servers on 20th May 2014.

So the main feature is to dump the packets of a tcpdump (done by tcpdump or wireshark) in a user-friendly format. After first experiments, it becomes also a live racing statistic tool.

Terms and definition

Template:MKWii Network Terms

The Tool

Change log

mkw-ana v0.18 r1836 - 2014-03-24

 - New command: FRIENDCODES: A friend code calculator. For each scanned
   parameter a line with a decimal friend code, a hexadecimal friend code,
   a decimal player ID and a game ID is printed. As input, various formats
   are supported. The new option --gid defines a game ID for creating a FC
   from a player id.

 - New command: ANALYZE: Analyze the dump files for special information and
   print it a machine readable list, one line per record. The first paraneter
   a comma separated list of keywords: PARAM-NAMES, NICKS, ALL.
   See built-in help for details.

 - New letter ID scheme for clients. All IDs are used circulary:
    0-9: Known Nintendo servers (detected by a previous DNS query).
    A-Z: Home client (--ip or --wii or atuo detect).
    a-z: Other clients (players).

 - Internal statistics about UDP (old) and TCP (new) traffic (send+receive
   packtes+bytes). The client dumps print now a summary of TCP+UDP.

 - Command 'CLIENT' prints the clients in order of appearance. If option
   --long is set, a multi line list with more details is printed.

 - The new command SERVERS prints a multi line list like 'CLIENTS --long',
   but only Nintendos servers are selected.

 - Command COUNT will now count also ARP, DNS and TCP records.

 - All commands can be prefixed by a 'C' (e.g. 'CDUMP' instead of 'DUMP') to
   force colorized output (= force option --color).

 - Options --ip and --wii accept now an optional port number.

 - Bug fix for 'A' detection of DNS replies.


mkw-ana v0.17 r1786 - 2014-03-03

 - New modes for option --print:
     DELAY :  Print the delay in ms (difference between racing and real time).
     LAST  :  Timeout since last packet in ms.
     XTIME := TIME,DELAY,LAST

 - New record type: QUIT: It is send, if a player quits the game.
   If --logmode=TEST is set, a log message is printed if QUIT occurs.

 - Some minor optical and layout changes.
Old change log

Show text


Built-in Help

Let's start with the built-in help as an overview about the tool:

Show text


General Description

The tool started as simple hex dumper reading network dumps in PCAP format. In the first phase of the tool, the textual dumps of wireshark and tcpdump were much better. But after only a few days, the tool learned to handle records, clients, users, friend codes and Miis. From this moment the tool was better to analyze the Mario Kart Wii traffic.

Now, mkw-ana split the traffic into records and scans some data to detect stages of the online meeting. Stages are for example room, prepare race, count down, racing and end of race. It is able to separate races into events (grand prix and team rand prix) and to calculate racing tables. Racing data can also be exported to support live statistics.

At the moment there are three different kinds of hexdumps. All 3 are able to dump in one line mode to have large tables. Tool less is here a very good tool for vertical and horizontal scrolling. The stages are includes into the dump as comment lines. The dumped records can be filters by sending, receiving, proxy, record types, stage types and packet length. It is also possible to select the dumped bytes by indices and ranges.

Include logs into output

Another feature is, that mkw-ana can read comment files. If making videos of the dumped meetings, you can write such comment file. Each line starts with a timestamp followed by a comment. VirtualDub[1] is a good tool for this job. Then you must synchronize the comment file with the network dump. The start of the first game ("GO" in the video) is a very good point for synchronization. Here is an example of a comment file (in german):

# All lines beginning with an '#' are comments and ignored.

# Sync:       timestamp of dump - timestamp of video
> 2014-03-07 18:33:31.583 +0100 - 0:08:09.667

0:00:00.000  Video Start
0:00:54.600  Enter WFC first time
...
0:08:09.667  GO!   GP 1.1

Notes:

  • The first line is the real time of the start of the race minus the video time stamp. This is the synchronisation. An synchronisation can be done multiple times.
  • The lines with the video timestamp and comment follow.
  • The name of the comment file must be the same as the network dump, but it must have the extension ".info" instead of ".eth".
  • Here you can find example dumps and log files: http://download.wiimm.de/mkw-ana/dumps/

Download

You can find the latest and some old distributions here
Content
  • Binaries for:
    • Linux i386
    • Linux x86_64
    • Cygwin/Windows (Needed Cygwin[2] DLL files are delivered. Best is to install a Cygwin system).
  • Some scripts as examples.
  • Some BMG text examples.
  • Some doc files.
Sometimes I upload single tool updates (beta versions) for testers

Capture the network data

First you must capture the network traffic of the Wii. Therefore you must redirect it to a PC running a capture software. There are 3 general ways to to this:

  • If you have a manageable switch, enable port mirroring and send all Wii traffic to a PC.
  • Use your PC as router.
  • Use old network hubs (not switches). A hub will mirror all traffic of all ports to all others; it's just a multi port repeater and will slow down your network.

Use a software like tcpdump or wireshark to capture the data. Best is to save the captured data directly to a file or to send it to other commands like mkw-ana for a live analysis.

It's also possible to save the data to a file and to make a live analysis at the same time. Use the following command pipe:

tcpdump -w- -U -i eth1 host wii | tee save.dump | mkw-ana ...

It is important to filter the data by host ip_or_name, because foreign traffic interfere the wii traffic analysis and will have curious side effects.

If using wireshark, save the dump to a file and use the following command for a live analysis:

mkw-ana --follow DUMPFILE ...

See »Dumping Network Traffic« more more details.

Accepted file formats

mkw-ana accepts the following file formats for the network dumps:

  • PCAP 2.4 : Standard packet capturing file format[3].
    • Big and little endian are supported.
    • Timestamps in micro- and in nanoseconds are supported.
  • PCAP 2.4.modified : Like PCAP, but with an extend packet header. This format is used by several routers, AVM FRITZ!Box[4] is one example.
    • Big and little endian are supported.
    • Only microseconds timestamps are supported.
  • PCAP-NG 2.4.modified : A next generation (NG) PCAP format[5].
    • Big and little endian are supported.
    • Microseconds timestamps are assumed, other are not supported.
    • Only the Enhanced Packet Block[6] is supported to retrieve packets.
  • BZIP2 compression
    • mkw-ana detects a BZIP2 compression automatically. It is supported for all other dump file formats.

mkw-ana accpets any list of dump files. The file format is detected for each single input file, so mixed formats are possible. The special file name »-« (minus sign) means: Don't open the file and and read the standard input (stdin) instead. So one of the input files can be read via pipe.

Live Statistics

mkw-ana can scan the network traffic in real time and can produce makedoc or php data files. Together with ssh and an cgi script, a live statistic is created. Live means that the tables are updates 2-5 seconds after the race have finished.

How it works

The whole job is done by 3 processes:

  1. First, you must capture the network traffic like described above. Then use one of the commands:
    ... | mkw-ana log --md  DATAFILE
    ... | mkw-ana log --php DATAFILE
    mkw-ana --follow DUMPFILE log --md  DATAFILE
    mkw-ana --follow DUMPFILE log --php DATAFILE
    
  2. Each time, a new DATAFILE is written, it must be transferred to the web server. A script using ssh, sftp scp or ftp within an endless loop will do this job automatically.
  3. Last not least, a CGI or PHP script running at the web server must read the data files to serve a html-page to the visitors.

To see, what live means, visit the live statistics on Wednesday or Thursday between 19:10 and 20:30 CET (Central European Time).

Links

References

  1. Wikipedia: VirtualDub, a free video utility.
  2. Cygwin, a Linux like environment for Windows.
  3. Wireshark: LIBPCAP file format
  4. AVM FRITZ!Box, a widely used router in Germany
  5. Wincap: The next generation PCAP file format
  6. Wincap: PCAP-NG, Enhanced Packet Block


Template:MKWii Network Protocol