Forum statistics, which aircraft is most popular

Discussion in 'OFF-Topic / Misc.' started by Marcel, Jan 7, 2017.

  1. Marcel

    Marcel Well-Known Member

    Joined:
    Sep 19, 2006
    Messages:
    7,045
    Likes Received:
    599
    Trophy Points:
    113
    Gender:
    Male
    Occupation:
    Bioinformatician
    Location:
    Dordrecht
    There was a question on the forum about bots, which reminded me of a little experiment I did some years ago. I was at that time preparing for a project at my work where I would mine data from scientific publications and do some analysis on that. So I thought as a first experiment to do the same on this forum. So I wrote a bot that mined the posts from the Aviation forum and did some simple analysis on it. The thread you can find here The forum's most discussed single engined fighter of ww2 is....

    So after reading the post about bots, I decided to brush off the dust on that old script and see if it would work again on the new forum layout. Of course it didn't. So I cleaned up the code, migrated the code from python2 to python3 (the program language, not the snake) and enhanced the working of the script.

    The old script was very simple. It collected all the posts, filtered out the quotes and stored it as one long text file. I now changed this behavior in that it also stores the author, the posted date and the post number in the thread. The mined data will then be put in a little relational database so I can analyse more easily. I did not include the word-cloud script anymore, can still add that later.

    For the programmers among us, the code can be found on github: GitHub - mke21/ww2aircraftcrawler: little script to find some simple statistics on the ww2aircraft forum.

    If you didn't follow all that technical stuff, the summary is this: I made a script to count the occurrence of aircraft names on the forum. So I can count how many posts contained the word 'spitfire' or another aircraft. I can also do that per user or per year. I thought it would be a nice experiment to think up some analysis that I can do. These are the same techniques that Google, Microsoft and Facebook release on you in your daily browsing time. Of course their setup is much more sophisticated
    For now I have a list of 90 single engine fighters. I will post this later in a post. Maybe you guys can come up with ideas that we could try.


    I will post the current list tomorrow. Hope you will join me.
     
    • Informative Informative x 1
  2. Wurger

    Wurger Siggy Master
    Staff Member Administrator

    Joined:
    Jun 19, 2005
    Messages:
    49,578
    Likes Received:
    2,323
    Trophy Points:
    113
    Occupation:
    A retired military Navigator/ATC, FIS controller
    Location:
    Poland
  3. Gnomey

    Gnomey World Travelling Doctor
    Staff Member Moderator

    Joined:
    Nov 28, 2004
    Messages:
    42,500
    Likes Received:
    570
    Trophy Points:
    113
    Gender:
    Male
    Occupation:
    Doctor
    Location:
    Portsmouth / Royal Deeside, UK
    Home Page:
    Very cool Marcel, will be interesting to see the results.
     
  4. Wayne Little

    Wayne Little Well-Known Member

    Joined:
    Oct 7, 2006
    Messages:
    51,903
    Likes Received:
    887
    Trophy Points:
    113
    Location:
    Adelaide Sth. Aust.
    Bring it on...
     
  5. Marcel

    Marcel Well-Known Member

    Joined:
    Sep 19, 2006
    Messages:
    7,045
    Likes Received:
    599
    Trophy Points:
    113
    Gender:
    Male
    Occupation:
    Bioinformatician
    Location:
    Dordrecht
    No ideas? I'll post some next time to give you some ideas what I can do here.

    So here are the 90 aircraft that I already have. In the program I try to take into account all ways that a user could type the name. So a '-' can also be a '.' or a ' ' or a '_' etc. BF109 can also be ME109, Spitfire can be Spit etc. Making these search strings is most difficult and time consuming. I use regex for that, if you want, you can look the real search-strings in the git repository I gave in the opening post. You'll have to look at the file 'aircraft.py' which is in the ww2crawler directory.

    Code:
    Avia B-534
    Avia BH-33
    Armstrong Whitworth Scimitar
    Boeing P-12
    Bristol Bulldog
    Fairey Firefly II
    Fiat CR.32
    Fiat CR.42
    Gloster Gamecock II
    Gloster Gauntlet
    Gloster Gladiator
    Grumman F3F
    Hawker Demon
    Hawker Fury
    Hawker Nimrod
    Heinkel He 51
    Kawasaki Ki-10
    Koolhoven F.K.52
    Polikarpov I-15
    Arsenal VG-33
    Avia B-135
    Bell P-39 Aircobra
    Bell P-63 Kingcobra
    Bloch MB.150-157
    Boeing P-26 Peashooter
    Breda Ba.27
    Brewster Buffalo
    CAC Boomerang
    Caudron C.714
    Curtiss P-36
    Curtiss P-40
    Curtiss-Wright CW-21
    Dewoitine D.500/D.510
    Dewoitine D.520
    Fiat G.50
    Fiat G.55
    Focke-Wulf Fw 190
    Focke-Wulf Ta 152
    Fokker D.XXI
    Grumman F4F/FM Wildcat
    Grumman F6F Hellcat
    Grumman F8F Bearcat
    Hawker Hurricane
    Hawker Tempest
    Hawker Typhoon
    Heinkel He 112
    IAR 80
    Ikarus IK-2
    Kawanishi N1K/N1K-J
    Kawasaki Ki-61
    Kawasaki Ki-100
    Koolhoven F.K.58
    Lavochkin LaGG-1
    Lavochkin LaGG-3
    Lavochkin La-5
    Lavochkin La-7
    Loire 46
    Macchi C.200
    Macchi C.202
    Macchi C.205
    Messerschmitt Bf 109
    Mikoyan-Gurevich MiG-1
    Mikoyan-Gurevich MiG-3
    Mitsubishi A5M
    Mitsubishi A6M Zero
    Mitsubishi J2M
    Morane-Saulnier M.S.406
    Nakajima Ki-27
    Nakajima Ki-43
    Nakajima Ki-44
    Nakajima Ki-84
    North American P-51 Mustang
    Polikarpov I-16
    PZL P.7
    PZL P.11
    PZL P.24
    Reggiane Re.2000
    Reggiane Re.2001
    Reggiane Re.2005
    Republic P-43
    Republic P-47 Thunderbolt
    Seversky P-35
    Supermarine Seafire
    Supermarine Spitfire
    Vought F4U/FG Corsair
    Vultee P-66 Vanguard
    Yakovlev Yak-1
    Yakovlev Yak-3
    Yakovlev Yak-7
    Yakovlev Yak-9
    
     
  6. Marcel

    Marcel Well-Known Member

    Joined:
    Sep 19, 2006
    Messages:
    7,045
    Likes Received:
    599
    Trophy Points:
    113
    Gender:
    Male
    Occupation:
    Bioinformatician
    Location:
    Dordrecht
    #6 Marcel, Jan 8, 2017
    Last edited: Jan 8, 2017
    So what can you do. Mind you this is a simple script I wrote. I am not using any 'Natural Language' or other fancy algorithms, this is simply a wordcount. Also the list of aircraft is not extensive as you saw.
    Yesterday evening, I set the script to work to download all of the 6000+ threads in the Aviation forum. For me this forum represents the hart of this site. This downloading is by fat the longest step here, as crawling through this forum takes about 3 hours when using a single cpu. Multithreading would make this faster, but that would make the script more complex and I don't want to spend money hiring an Amazon cluster.

    First of all, the most simple of questions: who is the champion mentioning SE fighters in his post? This is the top 10:
    Code:
    Shortround6|2937
    tomo pauk|2630
    GregP|2307
    drgondog|2089
    stona|1587
    davebender|1302
    FLYBOYJ|1210
    parsifal|1160
    wuzak|980
    kool kitty89|961
    
    The other obvious one, which aircraft of the list is mentioned most in the posts. Although I am now only counting the post and mentioning twice in one post still only counts as one, I think not much have changed since the last time I did this:
    Code:
    Supermarine Spitfire|16436
    North American P-51 Mustang|12972
    Messerschmitt Bf 109|8848
    Republic P-47 Thunderbolt|7446
    Focke-Wulf Fw 190|7280
    Curtiss P-40|5909
    Hawker Hurricane|5742
    Mitsubishi A6M Zero|5575
    Vought F4U/FG Corsair|5347
    Grumman F6F Hellcat|3313
    Grumman F4F/FM Wildcat|3219
    Bell P-39 Aircobra|2948
    Focke-Wulf Ta 152|2130
    Hawker Typhoon|2056
    Hawker Tempest|1772
    Brewster Buffalo|1596
    Grumman F8F Bearcat|1124
    Hawker Demon|1065
    Curtiss P-36|926
    Bell P-63 Kingcobra|824
    Supermarine Seafire|743
    Mikoyan-Gurevich MiG-1|646
    Hawker Fury|627
    Nakajima Ki-43|627
    Fokker D.XXI|615
    Gloster Gladiator|555
    Heinkel He 51|515
    Polikarpov I-16|510
    Nakajima Ki-84|498
    Lavochkin La-5|467
    Kawasaki Ki-61|454
    Yakovlev Yak-3|438
    Fiat G.50|420
    Fiat G.55|408
    Lavochkin La-7|386
    Dewoitine D.500/D.510|360
    Heinkel He 112|331
    Mikoyan-Gurevich MiG-3|322
    Fiat CR.42|316
    Republic P-43|308
    Dewoitine D.520|307
    Yakovlev Yak-1|307
    Seversky P-35|279
    Reggiane Re.2005|258
    Kawasaki Ki-100|251
    Nakajima Ki-44|245
    Lavochkin LaGG-3|242
    Nakajima Ki-27|236
    Mitsubishi J2M|213
    Boeing P-26 Peashooter|200
    Morane-Saulnier M.S.406|199
    Macchi C.202|197
    Polikarpov I-15|193
    CAC Boomerang|189
    Kawanishi N1K/N1K-J|184
    Macchi C.200|145
    Mitsubishi A5M|141
    Macchi C.205|132
    Vultee P-66 Vanguard|118
    Reggiane Re.2000|113
    Arsenal VG-33|96
    Curtiss-Wright CW-21|94
    Grumman F3F|81
    Reggiane Re.2001|80
    IAR 80|72
    Yakovlev Yak-7|69
    Yakovlev Yak-9|69
    Gloster Gauntlet|64
    Fiat CR.32|52
    Avia B-534|46
    Hawker Nimrod|40
    Bloch MB.150-157|30
    Lavochkin LaGG-1|26
    Avia B-135|22
    Boeing P-12|22
    Ikarus IK-2|18
    PZL P.11|18
    Koolhoven F.K.58|17
    Caudron C.714|14
    Armstrong Whitworth Scimitar|13
    Kawasaki Ki-10|13
    Loire 46|13
    PZL P.24|6
    PZL P.7|4
    Breda Ba.27|3
    Koolhoven F.K.52|3
    Fairey Firefly II|1
    Gloster Gamecock II|1
    
    I was right, the spitfire is on top again, so who is so fond of this spitfire? Again the top 10:
    Code:
    tomo pauk|836
    Shortround6|796
    stona|784
    wuzak|460
    GregP|438
    parsifal|424
    drgondog|410
    nuuumannn|367
    Glider|347
    Soren|310
    
    :lol: Soren in the top 10 and the oldies know for sure that Soren was not a real fan of the Spitfire. This illustrates the limitation of this methode. It doesn't tell you whether the post was positive or negative. For that I would have to put way more time into this.

    So what is my most mentioned aircraft in the Aviation forum?
    Code:
    Marcel|Fokker D.XXI|75
    Marcel|Messerschmitt Bf 109|43
    Marcel|Supermarine Spitfire|33
    Marcel|Brewster Buffalo|26
    Marcel|Hawker Hurricane|18
    Marcel|Heinkel He 112|17
    Marcel|North American P-51 Mustang|10
    Marcel|Focke-Wulf Fw 190|7
    Marcel|Mitsubishi A6M Zero|6
    Marcel|Vought F4U/FG Corsair|5
    Marcel|Grumman F4F/FM Wildcat|4
    Marcel|Koolhoven F.K.58|4
    Marcel|Curtiss P-36|3
    Marcel|Curtiss P-40|3
    Marcel|Grumman F6F Hellcat|3
    Marcel|IAR 80|3
    Marcel|Bell P-39 Aircobra|2
    Marcel|Curtiss-Wright CW-21|2
    Marcel|Fiat G.50|2
    Marcel|Hawker Tempest|2
    Marcel|Breda Ba.27|1
    Marcel|Caudron C.714|1
    Marcel|Dewoitine D.520|1
    Marcel|Gloster Gamecock II|1
    Marcel|Gloster Gauntlet|1
    Marcel|Gloster Gladiator|1
    Marcel|Hawker Fury|1
    Marcel|PZL P.11|1
    Marcel|Polikarpov I-15|1
    Marcel|Polikarpov I-16|1
    
    Again no surprises there, the Fokker on top:) It is probably because I did the D.XXI in Dutch service thread, some years ago.
     
    • Bacon Bacon x 3
  7. Wurger

    Wurger Siggy Master
    Staff Member Administrator

    Joined:
    Jun 19, 2005
    Messages:
    49,578
    Likes Received:
    2,323
    Trophy Points:
    113
    Occupation:
    A retired military Navigator/ATC, FIS controller
    Location:
    Poland
  8. herman1rg

    herman1rg Well-Known Member

    Joined:
    Dec 3, 2008
    Messages:
    1,186
    Likes Received:
    80
    Trophy Points:
    48
    I think I might now what my own most mentioned aircraft is...................
     
  9. Marcel

    Marcel Well-Known Member

    Joined:
    Sep 19, 2006
    Messages:
    7,045
    Likes Received:
    599
    Trophy Points:
    113
    Gender:
    Male
    Occupation:
    Bioinformatician
    Location:
    Dordrecht
    When I start the computer again, I will see what it is.

    Btw are there any requests for aircraft? You see my current list in an earlier post but which aircraft do you want me to add?
     
    • Bacon Bacon x 1
  10. GrauGeist

    GrauGeist Well-Known Member

    Joined:
    Aug 29, 2008
    Messages:
    15,456
    Likes Received:
    2,167
    Trophy Points:
    113
    Occupation:
    Public Safety Automotive Technician
    Location:
    Redding, California
    Home Page:
    How about some obscure fighter types that saw service in WWII?

    Avia B-135 (Czechslovakia)
    MAVAG Heja/MÁVAG Héja (Hungary)
    Rogozarski/Rogožarski IK-3 (Yugoslavia)
    VL Myrsky (Finland)
     
  11. rochie

    rochie Well-Known Member

    Joined:
    Aug 9, 2006
    Messages:
    14,840
    Likes Received:
    458
    Trophy Points:
    83
    Gender:
    Male
    Occupation:
    Head chef
    Location:
    billingham nr middlesbrough uk
    Great stuff Marcel.

    So you have proved the Spitfire was the greatest fighter of all time and the Bf 109 was only third !

    :lol::lol::lol::lol::lol::lol::lol:
     
    • Like Like x 1
  12. parsifal

    parsifal Well-Known Member

    Joined:
    Apr 6, 2008
    Messages:
    10,836
    Likes Received:
    737
    Trophy Points:
    113
    Occupation:
    Urban Design/Strategic Studies Tutor
    Location:
    Orange NSW
    Very impressive.....
     
  13. GrauGeist

    GrauGeist Well-Known Member

    Joined:
    Aug 29, 2008
    Messages:
    15,456
    Likes Received:
    2,167
    Trophy Points:
    113
    Occupation:
    Public Safety Automotive Technician
    Location:
    Redding, California
    Home Page:
    Only these were the best, Karl, because they had those pesky roundels removed! :evil4:

    Spitfire_EN830.jpg
     
    • Like Like x 2
  14. Thorlifter

    Thorlifter Well-Known Member

    Joined:
    Jun 10, 2004
    Messages:
    7,978
    Likes Received:
    209
    Trophy Points:
    63
    Occupation:
    IT Nerd
    Location:
    Dallas, Tx Jubail, Saudi Arabia
    Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair Corsair

    There, that ought to help move the best plane of WWII to the top of the list!!! :lol:
     
  15. Marcel

    Marcel Well-Known Member

    Joined:
    Sep 19, 2006
    Messages:
    7,045
    Likes Received:
    599
    Trophy Points:
    113
    Gender:
    Male
    Occupation:
    Bioinformatician
    Location:
    Dordrecht
    Said the man with a P40 in his siggy :lol:

    2 problems though:
    1. I only mined the Aviation forum
    2. All these corsairs are in 1 post, so count as 1

    :lol:
     
  16. rochie

    rochie Well-Known Member

    Joined:
    Aug 9, 2006
    Messages:
    14,840
    Likes Received:
    458
    Trophy Points:
    83
    Gender:
    Male
    Occupation:
    Head chef
    Location:
    billingham nr middlesbrough uk
    :lol::lol::lol::lol::lol::lol::lol::lol:
     
  17. Thorlifter

    Thorlifter Well-Known Member

    Joined:
    Jun 10, 2004
    Messages:
    7,978
    Likes Received:
    209
    Trophy Points:
    63
    Occupation:
    IT Nerd
    Location:
    Dallas, Tx Jubail, Saudi Arabia
    upload_2017-1-9_15-27-27.png
     
  18. Wurger

    Wurger Siggy Master
    Staff Member Administrator

    Joined:
    Jun 19, 2005
    Messages:
    49,578
    Likes Received:
    2,323
    Trophy Points:
    113
    Occupation:
    A retired military Navigator/ATC, FIS controller
    Location:
    Poland
  19. Marcel

    Marcel Well-Known Member

    Joined:
    Sep 19, 2006
    Messages:
    7,045
    Likes Received:
    599
    Trophy Points:
    113
    Gender:
    Male
    Occupation:
    Bioinformatician
    Location:
    Dordrecht
    Okay Herman, you don't talk much about SE fighters. Here is your score:
    Code:
    herman1rg|Hawker Hurricane|5
    herman1rg|Supermarine Spitfire|4
    herman1rg|Grumman F6F Hellcat|2
    herman1rg|Hawker Demon|2
    herman1rg|Messerschmitt Bf 109|2
    herman1rg|Republic P-47 Thunderbolt|2
    herman1rg|Grumman F4F/FM Wildcat|1
    herman1rg|Hawker Fury|1
    herman1rg|North American P-51 Mustang|1
    
     
  20. Marcel

    Marcel Well-Known Member

    Joined:
    Sep 19, 2006
    Messages:
    7,045
    Likes Received:
    599
    Trophy Points:
    113
    Gender:
    Male
    Occupation:
    Bioinformatician
    Location:
    Dordrecht
Loading...

Share This Page