Tuesday, 2 December 2014

Socorro: the Super Search Fields guide

Socorro has a master list of fields, called the Super Search Fields, that controls several parts of the application: Super Search and its derivatives (Signature report, Your crash reports... ), available columns in report/list/, and exposed fields in the public API. Fields contained in that list are known to the application, and have a set of attributes that define the behavior of the app regarding each of those fields. An explanation of those attributes can be found in our documentation.

In this guide, I will show you how to use the administration tool we built to manage that list.

You need to be a superuser to be able to use this administration tool.

Understanding the effects of this list

It is important to fully understand the effects of adding, removing or editing a field in this Super Search Fields tool.

A field needs to have a unique Name, and a unique combination of Namespace and Name in database. Those are the only mandatory values for a field. Thus, if a field does not define any other attribute and keeps their default values, it won't have any impact in the application -- it will merely be "known", that's all.

Now, here are the important attributes and their effects:

  • Is exposed - if this value is checked, the field will be accessible in Super Search as a filter.
  • Is returned - if this value is checked, the field will be accessible in Super Search as a facet / aggregation. It will also be available as a column in Super Search and report/list/, and it will be returned in the public API.
  • Permissions needed - permissions listed in this attribute will be required for a user to be able to use or see this field.
  • Storage mapping - this value will be used when creating the mapping to use in Elasticsearch. It changes the way the field is stored. You can use this value to define some special rules for a field, for example if it needs a specific analyzer. This is a sensitive attribute, if you don't know what to do with it, leave it empty and Elasticsearch will guess what the best mapping is for that field.

It is, as always, a rule of thumb to apply changes to the dev/staging environments before doing so in production. And to my Mozilla colleagues: this is mandatory! Please always apply any change to stage first, verify it works as you want (using Super Search for example), then apply it to production and verify there.

Getting there

To get to the Super Search Fields admin tool, you first need to be logged in as a superuser. Once that's done, you will see a link to the administration in the bottom-right corner of the page.

Fig 1 - Admin link

Clicking that link will get you to the admin home page, where you will find a link to the Super Search Fields page.

Fig 2 - Admin home page

The Super Search Fields page lists all the currently known fields with their attributes.

Fig 3 - Super Search Fields page

Adding a new field

On the Super Search Fields page, click the Create a new field link in the top-right corner. That leads you to a form.

Fig 4 - New field button

Fill all the inputs with the values you need. Note that Name is a unique identifier to this field, but also the name that will be displayed in Super Search. It doesn't have to be the same as Name in database. The current convention is to use the database name but in lower case and with underscores. So for example if your field is named DOMIPCEnabled in the database, we would make the Name something like dom_ipc_enabled.

Use the documentation about our attributes to understand how to fill that form.

Fig 5 - Example data for the new field form

Clicking the Create button might take some time, especially if you filled the Storage mapping attribute. If you did, in the back-end the application will perform a few things to very that this change does not break Elasticsearch indexing. If you get redirected to the Super Search Fields page, that means the operation was successful. Otherwise, an error will be displayed and you will need to press the Back button of your browser and fix the form data.

Note that cache is refreshed whenever you make a change to the list, so you can verify your changes right away by looking at the list.

Editing a field

Find the field you want to edit in the Super Search Fields list, and click the edit icon to the right of that field's row. That will lead you to a form much like the New field one, but prefilled with the current attributes' values of that field. Make the appropriate changes you need, and press the Update button. What applies to the New field form does apply here as well (mapping checks, cache refreshing, etc. ).

Fig 6 - The edit icon

Deleting a field

Find the field you want to edit in the Super Search Fields list, and click the delete icon to the right of that field's row. You will be prompted to confirm your intention. If you are sure about what you're doing, then confirm and you will be done.

Fig 7 - The delete icon

The missing fields tool

We have a tool that looks at all the fields known by Elasticsearch (meaning that Elasticsearch has received at least one document containing that field) and all the fields known in the Super Search Fields, and shows a diff of those. It is a good way to see if you did not forget some key fields that you could use in the app.

To access that list, click the See the list of missing fields link just above the Super Search Fields list.

Fig 8 - The missing fields link

The list of missing fields provides a direct link to create the field for each row. It will take you to the New field form with some prefilled values.

Fig 9 - Missing fields page

Conclusion

I think I have covered it all. If not, let me know and I'll adjust this guide. Same goes if you think some things are unclear or poorly explained.

If you find bugs in this Super Search Fields tool, please use Bugzilla to report them. And remember, Socorro is free / "libre" software, so you can also go ahead and fix the bugs yourself! :-)

Monday, 7 July 2014

Play Kawaii Killer, a horrible cuteness genocide game

Kawaii Killer, the first game of Tabemasu Games, is now out and playable on Android and iOS! Why am I writing about this game? Because it is the first commercial game of my friend Aurélien! He started as an independent game developer last year, and worked hard on making this Jam game a top-quality mobile game.

So what is this game about? You are a trapper going through his forest, and you want to kill as many animals as you can, as fast as possible. Each animal requires a different gesture to be killed, and it obviously gets more and more complex. If you are a sensitive person, or a friend of the animals, maybe you won't enjoy that game much, because those poor little things die in horrible manners… A lot of blood will be shed under your fingers! Be ready for a gory journey!

Maybe you have seen Kawaii Killer before: it was one of the best games created during the Game Dev Party Jam #3, back in 2012. Aurélien worked with a team of 5 people to build this very fun and enjoyable game. You can still find it here. Apart from the fact that the game has been rebuilt from scratch, much has been added since the Game Jam: new stages, new enemies, difficulty levels, a story, achievements and badges… If you have a smartphone and like killing cute animals, you should definitely try this game out. You can buy it from Google Play, the Apple App Store, or you can get it DRM-free from the Humble Store.

Not convinced yet? Well, take a look at the release trailer:

As a side note, I must say I am very happy that Aurélien, Fred and Vincent, the 3 guys who made this game, met during Jams organized by the Game Dev Party association. It's a proof that those events are useful to our community and help the video game industry in Lyon. Heck, Aurélien was an amateur game creator and now he has released a commercial game, isn't that awesome? So, let's keep rocking with the GDP, and let's keep building awesome video games in Lyon!

Tuesday, 1 July 2014

About Socorro

I have been working on Socorro for about 3 years now. It was about time that I started talking about it on this blog! With this first post, I am inaugurating 2 new tags, socorro and mozilla. I'm also going to add a language tag to my posts, so each new post will either be en or fr. This way, you can choose more accurately what kind of content you want to read from this blog. The previous links on the tags lead to the filtered Atom feeds, and I will add the mozilla feed to the Planet.

So, what is Socorro?

  1. a planet in the Star Wars universe
  2. a city in Brazil
  3. an island in Mexico
  4. a server to process breakpad crash reports

All of those are true, but as you guessed (I hope! ) we will only consider the last option. So basically, Socorro is a software that collects, processes and stores crash reports generated by breakpad. We use breakpad in our desktop software such as Firefox, Firefox for Android, Thunderbird and FirefoxOS. We then have a Web interface that shows information about the stability our those software, and we call it crash-stats. As usual with Mozilla, most of the information on this site is public, the only exceptions being sensitive data like email addresses, URLs, etc.

Most of Socorro's code base in written in Python. The back-end is raw Python based on a tool called configman, developed by my colleagues Lars and Peter, that handles the configuration and is used as a dependency injection system. It allows Socorro to be very modular, and to add or remove components easily without changing any code. We use 3 different databases:

  1. HBase is the primary storage and is there for data consistency (it contains all the data we receive and process and must never lose data)
  2. PostgreSQL is used as the main source of data for the UI (it contains all the processed data and has a lot of materialized views with various computed information)
  3. elasticsearch is used as the secondary source of data for the UI, mainly for everything related to search. This is what I have been mainly working on.

The Web app is based on Playdoh, Mozilla's customized django framework. We have more and more javascript, mostly using jQuery, but it's a bit chaotic at the moment.

Interested in contributing?

If Socorro sounds like a good project to you and you would like to participate, we would be very happy to hear about you! There are a few places where you can start:

You can also join us in Mozilla's IRC server, channel #breakpad. There are a good variety of bugs to fix on Socorro, from pure front-end work to complex back-end Python problems. And if Socorro is not your thing but you still want to help Mozilla fulfill its mission, you can take a look at whatcanidoformozilla.org!

Tuesday, 27 May 2014

Play Michel in Hell, our Ludum Dare 29 game!

Last year, my friend Aurélien invited me to participate with him and a few other friends in the Ludum Dare, for the 26th edition. We made a Flash game, which is probably why I didn't write about it (how could I face my Mozilla colleagues after making a Flash game? ). It's called Sniff and it's about a drug dealer that needs to deliver its magical powder to his customers. As those customers are in a night club, you will need to use the music to find your way through the town. Beware of the cops that want to catch you, and beware of not using the different powers the drug gives you...

Play Sniff

This blog post is not about Sniff, though. After having only organized, and not participated, in our last Game Dev Party Jam, I felt a bit frustrated and really wanted to work on a game. So I called a few friends and invited them to join me for the Ludum Dare 29 weekend. 4 of them answered, and we had a team! Caroline joined us as our graphics person, Pierre-Etienne would take care of all music things, while Maxime and Rémi would join me on the programming of the game.

michell.jpg

The tale of Michell

The theme of this edition of the Ludum Dare was "Beneath the surface". I was not much inspired by it, but I dislike multi-words themes in general. After about 3 hours of brain-storming, we decided to go for a tower defense game, with a bit of Hack 'n Slash in it. The idea was for the player to defend against troops sent from hell: Michell, the hero, peed on Lucifer's flowers, and that didn't make him happy at all. So Lucifer decided to send all his troops to kill him! But Michell is no mere hero, he is a magician that can grow strong plants to help him.

So, there you are on a small platform. Demons will be appearing around that platform in waves, and they will attack you. You can either fight them directly by clicking on them if you can reach them, but that is dangerous. Your other solution is to use your magic to grow powerful plants to help you. Click on a button at the bottom or use the shortcuts, then place your tower. The only limit is that you need to go close to where you want to build the tower, and wait until it is fully grown before you can do something else. The first tower is a bit weak and attacks enemies. The second one is more resilient, but it only slows them down.

Play Michell in Hell

This is the current version of the game. We continued developing the game after the end of the Ludum Dare, and have fixed some pretty big bugs. We also added more content (the second tower, the boss, sounds... ). If you want to see what the game looked like at the end of the weekend, you can play the Ludum Dare version.

The Ludum Dare 29 is now over, and we have the results of the voting. People liked our game, it seems, as we got only above-average marks. Unsurprisingly, the graphics and audio were what people liked the most! Overall we had slightly worst marks than for Sniff, but that doesn't surprise me: the game was less polished at the end of the weekend, with some annoying bugs. I hope people had played the updated version instead, but I realized only too late that I could edit the entry. Anyway, here are our ratings for the Jam competition!

ld29-ratings.png

Notes about Phaser

We used the Phaser game engine for this jam. I had played a little with it before, and I thought it was pretty cool and complete / stable enough. As usual, this assumption was stressed a lot during the weekend! Overall I am satisfied with it, the performance of the game is pretty good, the engine was quite easy to use, and the code of the engine itself is pretty clean. However, here are a few things that I didn't like or feel could be improved.

Documentation. That's the key for every developer facing code. If it isn't correctly documented, it's going to be very hard to use, people will get frustrated and the software will slowly die, or never get used at all. In the case of Phaser, discovery of features was the most difficult thing. I realized several times during the weekend that some things I had coded were already built inside the engine. That is frustrating, especially when coding in limited time like in a Jam. The API documentation was very complete and pretty good in my opinion, but it could do with some work on the search features.

Some things are still buggy, or at least did not behave like we expected them to. For example, we struggled a lot with the physics, and especially with the collision system. Sometimes collisions just stop working if you make objects unmovable. Objects push each others by default when colliding. And there are few incoherences in the collision functions that I would like to fix whenever I can find the time to open a pull request to Phaser...

The groups feature was the most misleading thing we used during the weekend. What we didn't know at first was that groups define the order in which objects are displayed. That's cool, but. If your Game object contains groups, those groups will be displayed in the order they are in Game. That's probably expected and a good behavior, but as we used groups for logical grouping (enemies together, towers together... ) and not for display grouping, we ended up with a messed up displaying. Basically, towers could never be behind enemies, even when that enemy was clearly in front of that tower. Same for the player. So I had to re-work our logic to not use Groups but lists, and use Groups as display layers. A bit confusing.

Overall, I was satisfied with Phaser and I want to use it again on my next Jams. Hopefully I will take some time to fix the few bugs we found during the development of Michell in Hell!

I want to finish this article with a big thank you to Caroline, Maxime, Pierre-Etienne and Rémi who made this game with me. I really enjoyed this weekend with you, and hopefully you did too!