Difference between revisions of "Wiimms mkw-ana (tool)"

From Custom Mario Kart
Jump to navigation Jump to search
(Bug fix for v1.03 is out.)
m
(21 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
{| class="textbox float-right grid"
 
{| class="textbox float-right grid"
 
|+ Wiimms mkw-ana
 
|+ Wiimms mkw-ana
| colspan=2 style="text-align:center" | [[file:Wiimms-SZS-Tools.png]]
+
| colspan=2 style="text-align:center" | [[File:Wiimms SZS Tools Logo.png]]
 
|-
 
|-
 
! Author:
 
! Author:
Line 16: Line 16:
 
|-
 
|-
 
! Current Version:
 
! Current Version:
| v1.03, 2015-06-27
+
| v1.15, 2017-08-21
 
|-
 
|-
 
! Homepage:
 
! Homepage:
 
| [http://mkw-ana.wiimm.de/ mkw-ana.wiimm.de]
 
| [http://mkw-ana.wiimm.de/ mkw-ana.wiimm.de]
 +
|-
 +
! Download:
 +
| [http://download.wiimm.de/mkw-ana/ download.wiimm.de]
 
|}
 
|}
  
The tools '''mkw-ana''' is a new project by [[Wiimm]] to analyze the network protocol of [[Mario Kart Wii]].
+
'''mkw-ana''' is a tool by [[Wiimm]] to analyze the network protocol of [[Mario Kart Wii]]. It can also print infos about current races and detect the usage of different cheats and used glitches.
  
  
Line 33: Line 36:
  
 
The main feature is to dump the packets of a tcpdump (done by ''tcpdump'' or ''wireshark'') in a user-friendly format. After first experiments, it also became a ''live racing statistic tool''.
 
The main feature is to dump the packets of a tcpdump (done by ''tcpdump'' or ''wireshark'') in a user-friendly format. After first experiments, it also became a ''live racing statistic tool''.
 
=== Terms and definition ===
 
 
{{MKWii Network Terms}}
 
  
 
== The Tool ==
 
== The Tool ==
  
Based on new knowledge made with Wiimmfi development, I have restructured mkw-ana.
+
Based on new knowledge made with [[Wiimmfi]] development, [[Wiimm]] has restructured '''mkw-ana'''. The important changes since v1.02 are:
The important changes are:
 
 
* New record analysis.
 
* New record analysis.
 
* Improved detection of race type (racing, balloon, coin, team, global, private room) and race stages.
 
* Improved detection of race type (racing, balloon, coin, team, global, private room) and race stages.
Line 54: Line 52:
 
** Support of an internal database to find the correct friend code calculation. This database can be modified by external files.
 
** Support of an internal database to find the correct friend code calculation. This database can be modified by external files.
 
* Support of an internal secret database. It is needed to decode MS answers. This database can be modified by external files.
 
* Support of an internal secret database. It is needed to decode MS answers. This database can be modified by external files.
 +
* User defined race and statistics tables. The columns can be enabled and ordered by the new option --table. Alternatively the table definitions can be exported by '''mkw-ana tables'', edited and then read by option --tfile.
 +
* Configurable tables: Columns can be selected and ordered by options or definiton files.
 +
* A backend server:
 +
** The backend server is enabled by option --cmd <ADDRESS>:<PORT> or --cmd unix:<FILE>.
 +
** The number of backend clients is not limited.
 +
** The backend server work either in a simple stream modus (use e.g. '''ncat''' or '''netcat''' for it) or in interactive modus using telnet negotiations.
 +
** The server has an built-in help. Just type "HELP".
 +
 +
With v1.07, mkw-ana can send ban jobs to Wiimmfi if a not allowed cheat is detected and the user have the rights to ban.
 +
* Using of 19 different glitches is checked. Some of them may trigger an autnoatic ban.
 +
* Many item cheats are detected. Because of to less packets false positives are possible. But if really an item cheat is used, the item cheat counter goes >4 in a race.
 +
* Instant drift is detected. A relative rate is used as trigger to avoid false positives.
 +
* Moving around before race start is detected.
 +
* Enabling an item before race start is detected.
 +
* Manipulation of finish time is detected.
 +
 +
With v1.13, the GPCM server send extra message to inform the cleint about room players and their status. mkw-ana extract these infos and creates a room table. But the new room status is not used at the frontend yet and only available at by beackend comamnd »WIIMMFI«. Try »WATCH WIIMMFI«.
  
 
=== Change log ===
 
=== Change log ===
  
 
<pre>
 
<pre>
mkw-ana v1.03 r2179 - 2015-06-27
+
mkw-ana v1.15 r2787 - 2017-08-21
  
  - Over the past few months I have collected new knowledge about status records.
+
  - Backend command »WIIMMFI« improved. Try »WATCH WIIMMFI«.
  And with this new knowledge I have re-implemented the client and race slot
 
  detection and also the event-type detection. Now global/worldwide events
 
  and also battles are correct interpreted. Overall, this is in progress.
 
  
  - Complete new client and racing slot analysis. It works now also for battles.
+
  - \wiimmfi\: New data about inactive clients scanned and used.
  
  - The RACE commands shows the racing table in the VIEW stage now.
+
  - New frontend command: BACKEND:
   Cheat detection is activated after a minimal delay.
+
   The frontend works like an interactive backend connection. (EXPERIMENTAL)
  
  - Messages (but not all) arrived during RACE table printing are printed in an
+
  - Different minor updates.
  own scroll region below the table without destroying the table itself. The
 
  scroll region size is calculated by the terminal height.
 
  
  - New option: --height=number: Define the terminal height (number of rows)
+
  - Windows only: Cygwin update to v2.8.1, 2017-07-03.
  and disable the automatic detection (like --width for the terminal width).
 
  
- New option for command RACE: --refresh=time: Defines the table refresh
 
  time (default is 0.5s).
 
  
- If a bad sized record is detected, then a warning is printed.
+
mkw-ana v1.14 r2631 - 2016-12-08
  
  - Command RACE supports now battles with some issues.
+
  - Detection of GPCM \wiimmfi\...\final\ packets.
 +
    - The statistics table is only available at the backend yet.
 +
    - Update of peer-connection flags.
  
  - New option: --assume=modes:
+
  - Bug fix: Command FLOWRATES failed sometimes.
    - RACE,BALLON,COIN: Interprete racing records always in the called method.
 
    - SLOT: Use the assumed record type for slot analysis.
 
  
  - Option --print=modes: 3 modes added:
+
  - Bug fix: Better detection of invalid STATUS packages. A warning is printed.
    - SLOT: Print client and race slot as first column.  
 
    - PID:  Print a column with the profile id.
 
    - TEAM: Print a column with the team name.
 
  
  - Option --log-mode=modes: 9 modes added:
+
  - New option: --warn-mode=LIST: Define, which warnings are enabled:
     - STAGE:  Print STATUS messages only on stage change.
+
     - LENGTH  Racing records with wrong length.
    - SLOTS:  Print logging lines for slot relevant changes.
+
     - XLENGTH :  Like RACE but be more verbose.
    - USLOTSPrint a refreshing table with used client and racing slots.
+
     - STATUS  : Invalid STATUS records.
              Try: mkw-ana log --lmd uslots DUMPFILE
+
     - EVENT  : Unknown event found.
     - RSLOTS:  Like as USLOTS, but without refreshing. Command RACE handles
+
     - DEFAULT := LENGTH | XLENGTH | STATUS
              USLOTS and RSLOTS in the same, but special way.
+
     - ALL and NONE are also possible.
     - ETYPE:   Log changes of the event-types (racing, battle, global, ...).
 
     - HOST:   Log host changes.
 
     - USER:   Log user definitions and slot assignments.
 
     - TEAM:    Log team definitions.
 
    - DISCONNECT: Log disconnects.
 
  
  - Command FC supports now NDS checksums. The checksum algorithm is selected
+
  - Command SILENT is now silent again. It also disables all warnings.
  by an internal game list. It can be overridden by new option --cmode=mode.
 
  Overall, the kind of working and list generation has changed.
 
 
 
- The dump commands try to detect the current game to use the correct friend
 
  code calculation.
 
 
 
- New command PID: Print a pid for each argument.
 
 
 
- Support of external files for game ID4 management:
 
    - New options: --gimport=file: Load the game list from a text file.
 
    - New options: --gexport=file: Save the game list to a text file.
 
    - Autoload of files /usr/local/share/mkw-ana/gameid.txt (global setup)
 
      and ~/.mkw-ana/gameid.txt (private setup).
 
 
 
- Command QUERY supports now imported and exported secret lists.
 
    - New options: --simport=file: Load the secret list from a text file.
 
    - New options: --sexport=file: Save the secret list to a text file.
 
    - Autoload of files /usr/local/share/mkw-ana/secret.txt (global setup)
 
      and ~/.mkw-ana/secret.txt (private setup).
 
 
 
- New command UPDATE: Updates the Game-ID and the SECRET lists. It works only
 
  if tool 'wget' is available.
 
 
 
- New color management.
 
 
 
- Option --dwc is not longer supported (network address changes frequently).
 
  Use --server=ip or --server=domain instead.
 
 
</pre>
 
</pre>
  
Line 143: Line 115:
 
The tool started as simple hex dumper reading network dumps in PCAP format. In the first phase of the tool, the textual dumps of ''wireshark'' and ''tcpdump'' were much better. But after only a few days, the tool learned to handle records, clients, users, friend codes and Miis. From this moment the tool was better to analyze the [[Mario Kart Wii]] traffic.
 
The tool started as simple hex dumper reading network dumps in PCAP format. In the first phase of the tool, the textual dumps of ''wireshark'' and ''tcpdump'' were much better. But after only a few days, the tool learned to handle records, clients, users, friend codes and Miis. From this moment the tool was better to analyze the [[Mario Kart Wii]] traffic.
  
Now, mkw-ana split the traffic into records and scans some data to detect stages of the online meeting. Stages are for example ''room'', ''prepare race'', ''count down'', ''racing'' and ''end of race''. It is able to separate races into events (''grand prix'' and ''team grand prix'') and to calculate racing tables. Racing data can also be exported to support live statistics.
+
Now, mkw-ana split the traffic into records and scans some data to detect stages of the online meetings. Stages are for example ''room'', ''prepare race'', ''count down'', ''racing'' and ''end of race''. It is able to separate races into events (''grand prix'' and ''team grand prix'') and to calculate racing tables. Racing data can also be exported to support live statistics.
  
At the moment there are three different kinds of hexdumps. All 3 are able to dump in one line mode to have large tables. Tool ''less'' is here a very good tool for vertical and horizontal scrolling. The stages are includes into the dump as comment lines. The dumped records can be filters by sending, receiving, proxy, record types, stage types and packet length. It is also possible to select the dumped bytes by indices and ranges.
+
At the moment there are four different kinds of hexdumps. All 4 are able to dump in one line mode to have large tables. Tool ''less'' is here a very good tool for vertical and horizontal scrolling. The stages are includes into the dump as comment lines. The dumped records can be filters by sending, receiving, proxy, record types, stage types and packet length. It is also possible to select the dumped bytes by indices and ranges.
  
=== Include logs into output ===
+
With beginning of version 1.02, [[Wiimm]] used the knowledge of the [[Wiimmfi]] development to re-implement record and stage detection. For example: If you enter a game in visitor modus, then mkw-ana shows racing data before you can see anything of the race on the screen.
  
Another feature is, that ''mkw-ana'' can read comment files. If making videos of the dumped meetings, you can write such comment file. Each line starts with a timestamp followed by a comment. VirtualDub<ref>Wikipedia: [http://en.wikipedia.org/wiki/VirtualDub VirtualDub], a free video utility.</ref> is a good tool for this job. Then you must synchronize the comment file with the network dump. The start of the first game ("GO" in the video) is a very good point for synchronization. Here is an example of a comment file (in german):
+
Since version 1.06, mkw-ana has a built-in backend server. The user can login with any telnet client and can e.g. reconfigure the table layout or request tables of old races or control the autoban system.
<pre>
 
# All lines beginning with an '#' are comments and ignored.
 
 
 
# Sync:      timestamp of dump - timestamp of video
 
> 2014-03-07 18:33:31.583 +0100 - 0:08:09.667
 
 
 
0:00:00.000  Video Start
 
0:00:54.600  Enter WFC first time
 
...
 
0:08:09.667  GO!  GP 1.1
 
</pre>
 
'''Notes:'''
 
* The first line is the real time of the start of the race minus the video time stamp. This is the synchronisation. An synchronisation can be done multiple times.
 
* The lines with the video timestamp and comment follow.
 
* The name of the comment file must be the same as the network dump, but it must have the extension ".info" instead of ".eth".
 
* Here you can find example dumps and log files: http://download.wiimm.de/mkw-ana/dumps/
 
  
 
=== Download ===
 
=== Download ===
Line 199: Line 155:
 
If using wireshark, save the dump to a file and use the following command for a live analysis:
 
If using wireshark, save the dump to a file and use the following command for a live analysis:
 
  mkw-ana COMMAND --follow DUMPFILE ...
 
  mkw-ana COMMAND --follow DUMPFILE ...
 +
DUMPFILE can not only be a standard file, it can also be a socket or a ZCP7IP connection.
  
 
See »[[Dumping Network Traffic]]« more more details.
 
See »[[Dumping Network Traffic]]« more more details.
Line 222: Line 179:
 
** ''mkw-ana'' detects a BZIP2 compression automatically. It is supported for all other dump file formats.
 
** ''mkw-ana'' detects a BZIP2 compression automatically. It is supported for all other dump file formats.
  
''mkw-ana'' accpets any list of dump files. The file format is detected for each single input file, so mixed formats are possible. The special file name »-« (minus sign) means: Don't open the file and and read the standard input (stdin) instead. So one of the input files can be read via pipe.
+
''mkw-ana'' accepts any list of dump files. The file format is detected for each single input file, so mixed formats are possible. The special file name »-« (minus sign) means: Don't open the file and and read the standard input (stdin) instead. So one of the input files can be read via pipe.
  
 
== Live Statistics ==
 
== Live Statistics ==
  
'''mkw-ana''' can scan the network traffic in real time and can produce ''makedoc'' or ''php'' data files. Together with '''ssh''' and an '''cgi script''', a [http://wiimm.de/mkw-race live statistic] is created. Live means that the tables are updates 2-5 seconds after the race have finished.
+
'''mkw-ana''' can scan the network traffic in real time print a complete race statistics and more at the text screen. Moreover it can create ''makedoc'' or ''php'' data files after each race to create.
 
 
=== How it works ===
 
  
The whole job is done by 3 processes:
 
<ol>
 
<li>
 
 
First, you must capture the network traffic like described above. Then use one of the commands:
 
First, you must capture the network traffic like described above. Then use one of the commands:
 
<pre>
 
<pre>
... | mkw-ana log --md  DATAFILE
+
... | mkw-ana race
... | mkw-ana log --php DATAFILE
+
mkw-ana race --follow DUMPFILE
mkw-ana --follow DUMPFILE log --md  DATAFILE
+
</pre> With this command you'll see information about selected the track, entering the room and a live game statistics. Important infos about cheats and glitches are added if detected.
mkw-ana --follow DUMPFILE log --php DATAFILE
 
</pre>
 
 
 
<li>
 
Each time, a new DATAFILE is written, it must be transferred to the web server. A script using '''ssh''', '''sftp''' '''scp''' or '''ftp''' within an endless loop will do this job automatically.
 
 
 
<li>
 
Last not least, a CGI or PHP script running at the web server must read the data files to serve a html-page to the visitors.
 
</ol>
 
 
 
To see, what live means, visit the [http://wiimm.de/mkw-race live statistics] on Wednesday or Thursday between 19:10 and 20:30 CET (Central European Time).
 
  
 
== Links ==
 
== Links ==
Line 260: Line 201:
 
<br/>
 
<br/>
 
{{MKWii Network Protocol}}
 
{{MKWii Network Protocol}}
[[Category:Software]]
+
[[Category:Software|M]]
[[category:Network Protocol]]
+
[[category:Network Protocol|M]]

Revision as of 10:38, 30 March 2021

Wiimms mkw-ana
Wiimms SZS Tools Logo.png
Author: Wiimm
Operating Systems: Linux (i386,x86_64),
Windows (Cygwin).
Software Type: Mario Kart Wii network traffic analyzer
File Formats: PCAP 2.4, PCAP 2.4.modified, PCAP-NG, bzip2 compression, BMG (text).
Current Version: v1.15, 2017-08-21
Homepage: mkw-ana.wiimm.de
Download: download.wiimm.de

mkw-ana is a tool by Wiimm to analyze the network protocol of Mario Kart Wii. It can also print infos about current races and detect the usage of different cheats and used glitches.



Intention

In November 2012, Wiimm decided to analyze the network protocol of Mario Kart Wii. The main goal was to discover online cheaters. Another goal was to set up his own server if Nintendo ever shut down its servers, which eventually happened on May 20th, 2014.

The main feature is to dump the packets of a tcpdump (done by tcpdump or wireshark) in a user-friendly format. After first experiments, it also became a live racing statistic tool.

The Tool

Based on new knowledge made with Wiimmfi development, Wiimm has restructured mkw-ana. The important changes since v1.02 are:

  • New record analysis.
  • Improved detection of race type (racing, balloon, coin, team, global, private room) and race stages.
  • Improved detection of current active players.
  • Improved hex dumps.
  • Complete new RACE command:
    • It shows battles and normal races.
    • It shows the in-game players while in VIEW mode.
    • Log messages are printed in an own scroll region.
    • special tables that shows client and racing slot usage.
  • Friend code calculation by Wii or by NDS mode.
    • Support of an internal database to find the correct friend code calculation. This database can be modified by external files.
  • Support of an internal secret database. It is needed to decode MS answers. This database can be modified by external files.
  • User defined race and statistics tables. The columns can be enabled and ordered by the new option --table. Alternatively the table definitions can be exported by 'mkw-ana tables, edited and then read by option --tfile.
  • Configurable tables: Columns can be selected and ordered by options or definiton files.
  • A backend server:
    • The backend server is enabled by option --cmd <ADDRESS>:<PORT> or --cmd unix:<FILE>.
    • The number of backend clients is not limited.
    • The backend server work either in a simple stream modus (use e.g. ncat or netcat for it) or in interactive modus using telnet negotiations.
    • The server has an built-in help. Just type "HELP".

With v1.07, mkw-ana can send ban jobs to Wiimmfi if a not allowed cheat is detected and the user have the rights to ban.

  • Using of 19 different glitches is checked. Some of them may trigger an autnoatic ban.
  • Many item cheats are detected. Because of to less packets false positives are possible. But if really an item cheat is used, the item cheat counter goes >4 in a race.
  • Instant drift is detected. A relative rate is used as trigger to avoid false positives.
  • Moving around before race start is detected.
  • Enabling an item before race start is detected.
  • Manipulation of finish time is detected.

With v1.13, the GPCM server send extra message to inform the cleint about room players and their status. mkw-ana extract these infos and creates a room table. But the new room status is not used at the frontend yet and only available at by beackend comamnd »WIIMMFI«. Try »WATCH WIIMMFI«.

Change log

mkw-ana v1.15 r2787 - 2017-08-21

 - Backend command »WIIMMFI« improved. Try »WATCH WIIMMFI«.

 - \wiimmfi\: New data about inactive clients scanned and used.

 - New frontend command: BACKEND:
   The frontend works like an interactive backend connection. (EXPERIMENTAL)

 - Different minor updates.

 - Windows only: Cygwin update to v2.8.1, 2017-07-03.


mkw-ana v1.14 r2631 - 2016-12-08

 - Detection of GPCM \wiimmfi\...\final\ packets.
    - The statistics table is only available at the backend yet.
    - Update of peer-connection flags.

 - Bug fix: Command FLOWRATES failed sometimes.

 - Bug fix: Better detection of invalid STATUS packages. A warning is printed.

 - New option: --warn-mode=LIST: Define, which warnings are enabled:
    - LENGTH  :  Racing records with wrong length.
    - XLENGTH :  Like RACE but be more verbose.
    - STATUS  :  Invalid STATUS records.
    - EVENT   :  Unknown event found.
    - DEFAULT := LENGTH | XLENGTH | STATUS
    - ALL and NONE are also possible.

 - Command SILENT is now silent again. It also disables all warnings.

old logs


General Description

The tool started as simple hex dumper reading network dumps in PCAP format. In the first phase of the tool, the textual dumps of wireshark and tcpdump were much better. But after only a few days, the tool learned to handle records, clients, users, friend codes and Miis. From this moment the tool was better to analyze the Mario Kart Wii traffic.

Now, mkw-ana split the traffic into records and scans some data to detect stages of the online meetings. Stages are for example room, prepare race, count down, racing and end of race. It is able to separate races into events (grand prix and team grand prix) and to calculate racing tables. Racing data can also be exported to support live statistics.

At the moment there are four different kinds of hexdumps. All 4 are able to dump in one line mode to have large tables. Tool less is here a very good tool for vertical and horizontal scrolling. The stages are includes into the dump as comment lines. The dumped records can be filters by sending, receiving, proxy, record types, stage types and packet length. It is also possible to select the dumped bytes by indices and ranges.

With beginning of version 1.02, Wiimm used the knowledge of the Wiimmfi development to re-implement record and stage detection. For example: If you enter a game in visitor modus, then mkw-ana shows racing data before you can see anything of the race on the screen.

Since version 1.06, mkw-ana has a built-in backend server. The user can login with any telnet client and can e.g. reconfigure the table layout or request tables of old races or control the autoban system.

Download

You can find the latest and some old distributions here
Content
  • Binaries for:
    • Linux i386
    • Linux x86_64
    • Cygwin/Windows (Needed Cygwin[1] DLL files are delivered. Best is to install a Cygwin system).
  • Some scripts as examples.
  • Some BMG text examples.
  • Some doc files.
Sometimes I upload single tool updates (beta versions) for testers

Capture the network data

First you must capture the network traffic of the Wii. Therefore you must redirect it to a PC running a capture software. There are 3 general ways to to this:

  • If you have a manageable switch, enable port mirroring and send all Wii traffic to a PC.
  • Use your PC as router.
  • Use old network hubs (not switches). A hub will mirror all traffic of all ports to all others; it's just a multi port repeater and will slow down your network.

Use a software like tcpdump or wireshark to capture the data. Best is to save the captured data directly to a file or to send it to other commands like mkw-ana for a live analysis.

It's also possible to save the data to a file and to make a live analysis at the same time. Use the following command pipe:

tcpdump -w- -U -i eth1 host wii | tee save.dump | mkw-ana ...

It is important to filter the data by host ip_or_name, because foreign traffic interfere the wii traffic analysis and will have curious side effects.

If using wireshark, save the dump to a file and use the following command for a live analysis:

mkw-ana COMMAND --follow DUMPFILE ...

DUMPFILE can not only be a standard file, it can also be a socket or a ZCP7IP connection.

See »Dumping Network Traffic« more more details.

Accepted file formats

mkw-ana accepts the following file formats for the network dumps:

  • PCAP 2.4 : Standard packet capturing file format[2].
    • Big and little endian are supported.
    • Timestamps in micro- and in nanoseconds are supported.
  • PCAP 2.4.modified : Like PCAP, but with an extend packet header. This format is used by several routers, AVM FRITZ!Box[3] is one example.
    • Big and little endian are supported.
    • Only microseconds timestamps are supported.
  • PCAP-NG 2.4.modified : A next generation (NG) PCAP format[4].
    • Big and little endian are supported.
    • Microseconds timestamps are assumed, other are not supported.
    • Only the Enhanced Packet Block[5] is supported to retrieve packets.
  • BZIP2 compression
    • mkw-ana detects a BZIP2 compression automatically. It is supported for all other dump file formats.

mkw-ana accepts any list of dump files. The file format is detected for each single input file, so mixed formats are possible. The special file name »-« (minus sign) means: Don't open the file and and read the standard input (stdin) instead. So one of the input files can be read via pipe.

Live Statistics

mkw-ana can scan the network traffic in real time print a complete race statistics and more at the text screen. Moreover it can create makedoc or php data files after each race to create.

First, you must capture the network traffic like described above. Then use one of the commands:

... | mkw-ana race
mkw-ana race --follow DUMPFILE

With this command you'll see information about selected the track, entering the room and a live game statistics. Important infos about cheats and glitches are added if detected.

Links

References

  1. Cygwin, a Linux like environment for Windows.
  2. Wireshark: LIBPCAP file format
  3. AVM FRITZ!Box, a widely used router in Germany
  4. Wincap: The next generation PCAP file format
  5. Wincap: PCAP-NG, Enhanced Packet Block


Template:MKWii Network Protocol