Orchard and Lucene: getting started with custom search – problems with booleans

by Oliver 12. September 2011 01:46

Update: I filed a bug in the Orchard issue tracker and it’s already been fixed. Nice work! We’ve just started to build our own customized search module. The goal is to have checkboxes for all searchable boolean properties of a given ContentPart and search by checking or unchecking them. In Orchard, one can add contents to the search index using the following code (this is an example for adding a boolean): 1: public class MyPartHandler : ContentHandler 2: { 3: public MyPartHandler(IRepository<MyPartRecord> repository) 4: { 5: Filters.Add(StorageFilter.For(repository)); 6:  7: OnIndexing<MyPart>( 8: (context, myPart) => context.DocumentIndex 9: .Add("DogsAllowed", myPart.DogsAllowed == true ? "true" : "false").Store()); 10: } 11: } Now, when a user checks the checkbox for “DogsAllowed” we want to ask the search index for all documents (Lucene term for what I would call an entity) with the DogsAllowed property set to true. The way we would do that is something along the following lines: 1: ... 2: var searchBuilder = _indexProvider().CreateSearchBuilder("Search") 3: searchBuilder.WithField("DogsAllowed", true); 4: ... The problem with this is: it gives me zero results! Of course, to get something into the index you have to go and save some document/entity for the OnIndexing handler to run, but I’d done that. Somehow I had the feeling that my search was not doing what it was supposed to… So I wanted to know more about the index Lucene had generated so far. Searching for lucene index visualizer lead me to Luke, the Lucene index toolbox which I downloaded v of because it was built against Lucene 2.9.1 and in Orchard the Lucene.NET version is only 2.9.2. Maybe a newer version would have also worked but I got what I wanted anyways. After providing the path to the index files, Luke showed me some interesting stuff, among other things that the boolean was stored as string “True” (mind the upper case!): This behavior is actually no surprise when we look at the source code for the Add(string name, bool value) method, it just calls the Add(string name, string value) method with the bool converted to a string: 1: public IDocumentIndex Add(string name, bool value) { 2: return Add(name, value.ToString()); 3: } It’s good to know that true.ToString() == “True” btw. So, now when I search inside Luke using the query DogsAllowed:True I do get a result – but using DogsAllowed:true I don’t! (The syntax for searches is field:value.)                So now, why does my search (in Orchard) for all documents that have the DogsAllowed property set to true return no results when there obviously is a document in the index with exactly that? Well, let’s look at the implementation of .WithField(): 1: public ISearchBuilder WithField(string field, bool value) { 2: CreatePendingClause(); 3: _query = new TermQuery(new Term(field, value.ToString().ToLower())); 4: return this; 5: } When I debugged through the search I saw that all of a sudden we were searching for DogsAllowed==true instead of True which to Lucene are two completely different things (at least in this scenario). Actually, turning on logging for the Lucene.Services namespace, we get some debug logging telling us what the search is currently looking for: 1: 2011-09-10 00:44:33,026 [6] Lucene.Services.LuceneIndexProvider - Searching: DogsAllowed:true* Now I’m not surprised anymore that I’m not getting any results! This is obviously a bug in Orchard’s Lucene module, but for now I can easily get around it using simply passing the lower case strings to the Add() method myself like this: 1: OnIndexing<MyPart>( 2: (context, myPart) => context.DocumentIndex 3: .Add("DogsAllowed", myPart.DogsAllowed == true ? "true" : "false").Store()); Happy coding!

Orchard CMS: module settings not visible in Admin area? placement.info is the key!

by Oliver 9. September 2011 22:29

Today I went off to create a custom search module for our new Orchard based web application. I simply copied the module Orchard.Search, renamed all namespaces and such to Teamaton.Search, replaced some strings and prepended a prefix of “Teamaton” to a bunch of them. I wanted the new module to be able to run side-by-side with the original Orchard.Search module to make sure I had an independent implementation and there wouldn’t be any conflicts in case the Orchard.Search module is already installed somewhere. A problem I ran into quite quickly was that I wouldn’t see any of the search index’s fields on the Admin page for the new module (to the right you see the settings in the original Search module which is what I expect to see on the left as well):             Searching the discussions on Orchard’s codeplex site, I found this very helpful post on Module site settings not showing up which promised to be the key to my problem. During the initial string replacement orgy, in SearchSettingsPartDriver I had changed return ContentShape(…).OnGroup(“search”) to return ContentShape(…).OnGroup(“TeamatonSearch”). Now, this wasn’t in sync anymore with the Group Id specified inside the SearchSettingsPartHandler, but it should be (as the mentioned post suggests). So I changed that to match the Group “TeamatonSearch” like so: 1: protected override void GetItemMetadata(GetContentItemMetadataContext context) { 2: if (context.ContentItem.ContentType != "Site") { 3: return; 4: } 5: base.GetItemMetadata(context); 6: context.Metadata.EditorGroupInfo.Add(new GroupInfo(T("TeamatonSearch"))); 7: } Unfortunately, it still did not work. Frustration. Another post on the discussion list led me to believe it was a problem with my Copy’n’Paste. There, piedone states: One remarkable point is a name clashing with content parts: there can't be content parts with the same name in different modules as this leads to problems with schema creation and also when selecting them in them admin menus (because then their displayed name is also the same). This somehow made sense to me, so I went and prefixed the part type name with Teamaton. When that didn’t work I also prefixed all properties of all ContentParts and Records because I remembered that I’d encountered a similar problem before when building our Google Maps module which, contained the same Latitude and Longitude property names that another ContentPart had already defined. But still: to no avail. Desperation. In the first post mentioned above, Bertrand Le Roy also states: I have been in that exact situation. This is probably an error getting swallowed somewhere. […] In my case, it was the feature not enabling all the classes I needed, in particular the record, and the system couldn't find how to map database records. So I went to the Orchard console … feature disable Teamaton.Search … feature enable Teamaton.Search … still nothing. Next it hit me like a sledge hammer – Migrations.cs! I hadn’t taken a single look at the persistence mapping for the SettingPartRecord! Well, turns out there was something wrong in there, but only since after I prefixed both parts and properties because now they wouldn’t match the strings in Migrations.cs anymore (a good example where magic strings break stuff). Kind of helpless, I launched NHProf to look at the database calls that were being made and saw that there weren’t any for the Teamaton.Search module – but for the Orchard.Search module there were … hm. I scanned through the project files again – and stared at placement.info, another file I hadn’t touched since copying the original Search module. Bingo! This was the root of all evil or rather of the search fields not displaying. It looked like this: 1: <Placement> 2: <Place Parts_Search_SiteSettings="Content:1"/> 3: <Place Parts_Search_SearchForm="Content:1"/> 4: </Placement> The problem with it was that I had renamed all the view files and prepended a prefix to them! So the correct version should look like this: 1: <Placement> 2: <Place Parts_TeamatonSearch_SiteSettings="Content:1"/> 3: <Place Parts_TeamatonSearch_SearchForm="Content:1"/> 4: </Placement> That’s because the views are named TeamatonSearch.SiteSettings.cshtml and TeamatonSearch.SearchForm.cshtml and are both placed inside a Parts folder: And now I saw the search fields even for our own new search module. Lesson learned If you can’t see something check the  placement.info  if you (accidentally) haven’t forgot to make that something visible And if you still can’t see it – double check! Happy coding!

Rendering problem with flot 0.7 and excanvas in IE8

by Oliver 8. September 2011 02:41

On Camping.info, we use flot 0.7 with jQuery and asynchronous updates for internal click statistics visualization. Unfortunately, we’ve been having some trouble with our flot graph – but only in IE8 and only sometimes! This has been really annoying, especially since we’re expecting a usage growth of this graph and IE8 has a user share of 25%. The problem is that after fetching the graph data from the server the graph sometime won’t appear on the canvas, although it seems to be there as the yellow tooltip suggests when hovering over the graph line. The graph looks like this (broken vs. intact):                      We were (are?) having a hard time reproducing this bug on our staging system but it would appear from time to time. On our live system we would get it quite often, although a page refresh often helped. Searching the web for flot+IE8 you’ll get a few results most of which point to the “flot-graphs” Google Group. A search on stackoverflow.com led me to an interesting post called IE8 is sometimes displaying my page correctly, sometimes not, which seemed interesting enough to check out. There, they talk about a problem with the  display:inline-block  CSS rule and that it is error-prone in IE<=8. So I checked the source of excanvas.min.js which is responsible for providing a canvas element to flot in IE browsers below version 9, and sure enough I found this: 1: a.cssText="canvas{display:inline-block;overflow:hidden;text-align:left;width:300px;height:150px} The fix proposed in this answer was to use zoom:1 to force elements with display:inline-block  to have a layout in IE (if you really want to know what that means, here’s more background info). Well, I tried that – to no avail. The next hint came from a post in the flot-graphs Google Group and proposed using flashcanvas instead of excanvas as the canvas provider for flot. Oh well, we haven’t got anything to lose, right? I replaced the script include and saw – exactly the same thing as before: nothing! (Check out the IE developer toolbar on the right – the you can see the Flash object inside the canvas element that was generated by flashcanvas.) What was left? Oh, I hadn’t tried to check for an updated version of excanvas yet – so I went and did that. The latest download version dates back to March 2009 but there has been some activity since then so I decide to go for the latest source which is also already a year and a half old but still by a whole year younger that the last official download I replaced the file and haven’t had any problem so far! We will be keeping an eye on this, but for now my hopes are high that this update might have finally helped. Happy coding!


by Oliver 17. August 2011 18:28

Lately, we encountered a problem with the speed of our search on www.camping.info for a certain set of search criteria. It sometimes used to take over a few seconds before the updated results were shown. This most likely seemed to be a problem with the database so I went to investigate the offending queries using the wonderful NHibernate Profiler. I found a very slow query: So I went and copied the long running query into a new query window in SSMS (SQL Server Management Studio). Since we use MS SQL Express on our production servers we don’t get the advanced database tuning advisor features of the full edition. Still, when you right click on any query window you’ll see the option “Include Actual Execution Plan” … [FULL]                                      [EXPRESS] … which already offers a lot of detail, as you can see in the following screenshot: When you right click on the execution plan you’ll see the option “Missing Index Details…” - there you get a CREATE INDEX statement that is ready to use once you give a name to the new index. I did this for three indexes and now we have this for the same query: 115 ms instead of 1993 ms – that’s an improvement of 94%! Even if DB queries lasting longer than 50-60 ms are not really fast anymore, we’ve still got quite an improvement here. Well, that’s it. Using a well-priced tool like NHibernate Profiler to identify slow queries and the Execution Plan feature of SSMS, we’re able to get quite a performance improvement in a short time. Happy Coding and Optimizing, Oliver

Running a multi-lingual web application: System.IO.PathTooLongException

by Oliver 4. March 2011 15:18

We definitely have long paths on our client’s platform www.camping.info, for example for a concrete rating on the detail page of a campsite with a long name in a state with a long name - http://www.camping.info/deutschland/schleswig-holstein-und-hamburg/campingplatz-ferien-und-campinganlage-schuldt-19573/bewertung/r23989 - but the path (everything after the .info including the ‘/’) is still only 112 characters long which is still a long way from the 260 character barrier that’s the default in ASP.NET (see the MSDN). The problem Well, the same page in Greek for example has the following URL: http://el.camping.info/γερμανία/σλέσβιχ-χολστάιν-αμβούργο/campingplatz-ferien-und-campinganlage-schuldt-19573/αξιολόγηση/r23989, at least that is what we see in the browser bar. Essentially, this URL will be encoded when requesting the page from the server so it becomes a gigantic http://el.camping.info/%CE%B3%CE%B5%CF%81%CE%BC%CE%B1%CE%BD%CE%AF%CE%B1/%CF%83%CE%BB%CE%AD%CF%83%CE%B2%CE%B9%CF%87-%CF%87%CE%BF%CE%BB%CF%83%CF%84%CE%AC%CE%B9%CE%BD-%CE%B1%CE%BC%CE%B2%CE%BF%CF%8D%CF%81%CE%B3%CE%BF/campingplatz-ferien-und-campinganlage-schuldt-19573/%CE%B1%CE%BE%CE%B9%CE%BF%CE%BB%CF%8C%CE%B3%CE%B7%CF%83%CE%B7/r23989! Now the path of the URL is a whopping 310 characters long! That’s quite a bit over the limit - but even for shorter URLs the cyrillic or greek equivalents surpass the limit not as rarely as one would think once they are URL encoded. The exception you get when exceeding the default of 260 chars is: System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be lessthan 260 characters, and the directory name must be less than 248 characters. And this is what the error looks like in Elmah: The solution You don’t have to search long to find a solution to this problem on the web: http://forums.asp.net/t/1553460.aspx/2/10. Just make sure the httpRuntime node contains the properties maxUrlLength AND relaxedUrlToFileSystemMapping like so: <httpRuntime maxUrlLength="500" relaxedUrlToFileSystemMapping="true" /> You might wonder what the relaxedUrlToFileSystemMapping property really does: you can read more on MSDN. In short, if set to true “the URL does not have to comply with Windows path rules” (MSDN). Happy coding, Oliver

Automatic deployment of an ASP.NET Web Application Project with TeamCity and MSBuild

by Oliver 21. January 2011 20:19

We recently updated one our largest project to use ASP.NET 4.0, and for this matter the new Package/Publish feature including sub-web.configs which is meant to supersede the Web Deployment Project. For a manual deployment there’s a good write-up on the msdn library titled ASP.NET Web Application Project Deployment Overview which shows how and where to set this up. In our case this was not satisfactory because our deployment process is a bit more complicated. We push our changes to a central repository and use JetBrains’ continuous integration server (CIS) TeamCity Professional, which is totally free for our project size, for a continuous integration process. Once TeamCity has pulled and tested the current version, it is supposed to deploy this version to our staging server where we further test the complete site. The key point in an automatic deployment was the management of the different web.config files for the different environments our project is running on. Unfortunately, until yesterday every deployment that included changes to the web.config file – even to the staging server - required a manual step of editing the web.config that live on our staging system (outside of source control!). What we used to do: after a successful build on our CIS we simply copied the web application (files) to our staging server! But as Scott Hanselman wrote: If You're Using XCopy, You're Doing It Wrong! This post inspired us to move along and take advantage of the new possibilities that we were given. In the meanwhile, before switching to .NET 4.0 actually, we also took a shot at the Web Deployment Project way of doing things but never actually got that far as to fully automate the deployment – somehow the setup was not as easy as we hoped. Anyway, we wanted web.config Transforms! So what does our setup look like and what did we want to do?   During local development and testing I use a web.config file that talks to a local DB instance and has some more specific settings. To run the web application on our staging server we need to replace certain values or whole sections in the web.config. For this transformation we use the sub-web.config files, one for each build configuration: Now, with all of these web.config files the simple XCOPY deployment we used to use does not work any longer. We need to trigger the web.config transformation on the build server and then deploy the whole application. As easy as this looks using the built-in menus and dialogs in Visual Studio – it took me quite a while to find how to do this in an automated build, more concretely from the command line. After unsuccessfulle skimming stackoverflow.com for a solution I finally tripped over this very informative blog post on publishing a VS2010 ASP.NET web application using MSBuild. Admittedly, the author focuses on how to publish on the local machine as it’s yet a different process but towards the end he posts the solution I was looking for: 1: msbuild Website.csproj "/p:Platform=AnyCPU;Configuration=Release;DesktopBuildPackageLocation=c:\_Publish\stage\Website.zip" /t:Package This was it! After running this on my machine with my own settings I looked into the folder with the zip file and found the following 5 files: At first I just wanted to take the zip file, copy it to the staging server, unpack it – done! But then I peaked into it… and deeper… and deeper… and… still deeper… until I finally saw our application files underneath this directory: This has got to be one of the longest paths I’ve ever seen and used! How would I automate the extraction of web application files from the zip with such a path? I was already seeing myself hacking away on the command line… But wait: what about those files that appeared next to the zip file? A ci-stage.deploy.cmd and a readme.txt caught my attention – of course, I opened the cmd file first :-D Well… maybe the readme file gives me a shortcut to understanding this and the rest of the 190 lines: Looks promising! I convinced myself to give it a shot. So we set up a new configuration in TeamCity with the following settings: These settings reflect the command line from above with a few minor changes (removed the DesktopBuildPackageLocation and set the /v[erbose] switch to m[inimal]): msbuild Website.csproj "/p:Platform=AnyCPU;Configuration=Release" /t:Package /v:m The second step is to use the generated script, ci-stage.deploy.cmd. I recommend to run the script by hand once using the /T switch just to make sure everything looks alright. In our case we found out that the deployment of the package would have deleted a lot of files, most of all images, from our website. This was not what we wanted! After a quick search I found this question on stackoverflow.com: MSDeploy: “Leave extra files on destination” from command line? So I added this switch to the parameters list in the build step configuration as follows: That’s it! This is all we need on the command line to generate a package that is ready for deployment on the staging server. There are a few more necessary settings, including the DesktopBuildPackageLocation, that can be found in the Package settings window inside the project properties of the web application project: the DesktopBuildPackageLocation can be set here instead of on the command line the website and application name on the destination IIS server that this package should be installed to some more options like excluding debug symbols etc. These settings are saved in the project file and will be used during generation and deployment of the package. That’s all I have to say right now. Happy Coding, Oliver

Javascript in UserControl in UpdatePanel

by Oliver 28. November 2009 15:17

Wenn man weiß, wie es geht, ist alles einfach. Aber JavaScript in UserControls innerhalb eines UpdatePanels funktionstüchtig zu bekommen hatte für mich - bis heute! - immer etwas mit Magie zu tun... Hinzu kam heute, dass ich noch Sys.UI.DomElement-Methoden nutzen wollte (aus der AJAX-Bibliothek). Folgende Versuche habe ich gemacht: 1. Page.ClientScript.RegisterClientScriptBlock(GetType(), "helpful_css_" + ClientID, script, true);2. ScriptManager.RegisterClientScriptBlock(this, GetType(), "helpful_" + ClientID, script, true);3. Page.ClientScript.RegisterStartupScript(GetType(), "helpful_css_" + ClientID, script, true);4. ScriptManager.RegisterStartupScript(this, GetType(), "helpful_css_" + ClientID, script, true); // <-- this is it!   Zu 1. und 3. ist zu sagen, dass sie innerhalb einen UpdatePanels nur nach dem ursprünglichen PageLoad funktionieren, nicht nach einem asynchronen Postback. Also fallen sie raus. Außerdem fallen 1. und 2. weg, weil an der Stelle, an der der hier übergebene JavaScript-Code in die Seite generiert wird, der $Sys-Namespace noch nicht definiert ist (ähnlich wie hier: http://encosia.com/2007/08/16/updated-your-webconfig-but-sys-is-still-undefined/). Bleibt nur 4. Und 4. funktioniert sowohl nach dem initialen PageLoad als auch nach einem async Postback und man kann $Sys verwenden. Wunderbar! 1. oder 2. bleiben allerdings nützlich, wenn man Custom Javascript in die Seite einfügen will, um es weiter unten auf der Seite zu nutzen. Wahlweise kann man natürlich auch die Methode RegisterClientScriptInclude nutzen, um das Javascript in eine eigene Datei auszulagern. Soviel dazu. Oliver

32-bit DLL in 64-bit WebApp: An attempt was made to load a program with an incorrect format

by Oliver 30. October 2009 16:36

Die folgende Fehlermeldung erhielt ich heute von unserem IIS7, als ich Camping.Info starten wollte: Server Error in '/' Application. Could not load file or assembly 'Microsoft.Cci' or one of its dependencies. An attempt was made to load a program with an incorrect format. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.BadImageFormatException: Could not load file or assembly 'Microsoft.Cci' or one of its dependencies. An attempt was made to load a program with an incorrect format. Also das Orakel gefragt und u.a. das hier gefunden: http://forums.asp.net/t/1358032.aspx Standardmäßig unterstützt ein 64-bittiger IIS 7 keine 32-bit-Module (u.a. DLLs). Man kann es ihm aber einfach beibringen :-) Im IIS-Manager den gewünschten ApplicationPool auswählen und in den Advanced Settings die folgende Einstellung vornehmen: Der Vollständigkeit halber hier noch ein Link zur Anleitung für den IIS6 auf Windows 2003 Server: http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/405f5bb5-87a3-43d2-8138-54b75db73aa1.mspx?mfr=true Happy Coding! Oliver

TDD: Testen ob eine Cookie gesetzt wurde

by robert 14. May 2009 16:14

Für Webentwickler ist das Auswerten und Setzen von Cookies Business-Logik und Business Logik sollte Testgetrieben entwickelt werden. Hier ein Beispiel Unit-Test: Response _response { get { return _httpCurrent.Response as ResponseNoWeb; } } RequestNoWeb _request { get { return _httpCurrent.Request as RequestNoWeb; } }   [Test] public void SetCookie() { _request.Url = new Uri("http://de.camping.info");   _uiLanguageService.SetLanguageCookie("de"); Assert.That(_response.CookiesCreated[0].Value, Is.EqualTo("de")); Assert.That(_response.CookiesCreated[0].Name, Is.EqualTo("Lang")); Assert.That(_response.CookiesCreated[0].Domain, Is.EqualTo(".camping.info")); }     Ermöglicht wird das über die Abstrakton von HttpRequest und HttpResponse. (Die Entwicklung wurde gestern begonnen, der Quelltext kapselt daher nur das Grundlegenste.)   Erfolgt die Anwendung nicht im Webkontext verwenden wir eine Implementierung für den lokalen Gebrauch:   namespace SpeakFriend.Utilities.Web { public class ResponseNoWeb : Response, IResponse { public void Redirect(string url) { _redirections.Add(url); }   public void Redirect(string url, bool endResponse) { _redirections.Add(url); }   public void SetCookie(HttpCookie cookie) { _cookiesCreated.Add(cookie); } } } …rudimentär, aber wirksam.

TranslatableUserControl in Repeater + TranslationPanel

by Oliver 24. March 2009 14:30

Nach zwei unabhängigen Fällen sieht es nun deutlicher danach aus, dass o.g. Kombination Probleme aufwirft und zwar mit dem EventHandling auf Serverseite. Genauer: Es werden keine Events für DropDownListen, LinkButtons u.ä. gefeuert, obwohl sie sollten. Das macht die Funktionalität im TranslationPanel kaputt.   Spannend ist folgendes Phänomen: Auf einer Seite mit der genannten Kombination und geöffnetem TranslationPanel geschieht folgendes: Erstes Mal die DDL benutzen -> nicht (optisch; im Fiddler sieht man aber einen Request) Zweites Mal die DDL benutzen -> die Seite wird neu geladen (komplett!) Drittes Mal die DDL benutzen -> funktioniert so, wie es soll Ab jetzt funzt es so wie es soll. Warum? Das also als Notiz an mich selbst. Mal schauen, was draus wird. Oli

About Oliver

shades-of-orange.com code blog logo I build web applications using ASP.NET and have a passion for javascript. Enjoy MVC 4 and Orchard CMS, and I do TDD whenever I can. I like clean code. Love to spend time with my wife and our children. My profile on Stack Exchange, a network of free, community-driven Q&A sites

About Anton

shades-of-orange.com code blog logo I'm a software developer at teamaton. I code in C# and work with MVC, Orchard, SpecFlow, Coypu and NHibernate. I enjoy beach volleyball, board games and Coke.