Glavs Blog

EOAST - Evolution of a software thingy - Part 2

In a part 1 of this series, I talked about what was the purpose of coming up with this "software thingy". Now I will tackle some of the design aspects that have evolved as I have been working on it.

Just to mention the requirements again though, it must be able to gather and store information from literally any source, and not tied to one collection mechanism. It must also be able to display any data in any format, typically something that the user/administrator configures to their liking. It must be easy to develop the collection and display parts. The main engine will take care of scheduling collections and storage of the data. Data can be pushed to it, not just collected. Must be reliable and "non blocking' in the event of a bad collection or display module.

It needs to allow me to exercise some of the new fandangled technologies but not at the expense of functionality.

Whew. Thats it in a nutshell. So what have I come up with so far (remember, being a pet project, its really a moving target). The diagram below shows a conceptual diagram of what needs to occur and the logical boundaries of components.

Obviously, a common element to house each item of information would be a good way to go initially. Something that caters for all types of information, with indicators pertaining to its source. As long as the data can be stored, and ideally searched, the display of that item is almost irrelevant for now.

Here is a sketching of the information items I initially wanted to capture/cater for.

So with the above things in mind, it seemed appropriate that I dub this piece of software the "Information Aggregator". Yes, like an RSS aggregator but with general information item in mind, which may or not be RSS items.

Continuing with the high level design though, extrapolating the above 2 diagrams further, and taking into account how I want the entire process to operate, below is an initial diagram of some components/services and how I expect interfaces and information flow to occur.

You can see that the "Engine" is the core component. The idea is that it maintains a list of all collection agents and display agents. The engine will co-ordinate scheduling of collections from the various agents, and also allow registered agents to push information to the engine.

Separately, upon receiving information or based on request from display agents, the engine will push information to any display agents, that have been associated with a particular collection type.

You can also see the interfaces that I have begun to design here. I will delve more into this in the next post, however the main interfaces being IPluginModule, IInformationPush and IInformationPull. IPluginModule being the interface that all compliant modules must implement. IInformationPush and IInformationPull are optional (although at least 1 makes sense) depending on the type of the module.

Note: I am aware you could probably use one of the composite application blocks to achieve an outcome similar to what I describe here, but I didn't want to rely on anything like that. Mainly because this is about learning and playing with stuff, as much as getting to an end solution.

So the eventual concept being built, is that information gets collected from some collection providers, lets say a provider called GlavsEmail and GlavsNoteEntry.

I have 3 registered display providers:

1. A balloon notification
2. A full detail display
3. A summary display.

I map the GlavsEmail information provider to the balloon notification and the summary display. I map the GlavsNoteEntry to the full detail display.

I set up the schedule for the GlavsEmail collection to occur every 1 hour. I set the schedule for the GlavsNoteEntry o be manually initiated (ie. by the information provider itself).

The engine should now perform the necessary collections when required, and when elements of GlavsEmail are gathered, display those using the 2 mapped display providers (balloon and summary), and the GlavsNoteEntry gets displayed using the full detail display.

At any time I can change this mapping. Also, all of the information from those sources is stored in SQL for easy searching and extraction if required. All I would typically have to do is register the collection and display providers, map them accordingly, and set up the access credentials for the collection provider (email in this case).

Thats it for this post. Next post I will delve more into the design of the interfaces, and technical details of the software (might be a few posts actually). Mind you, its still a moving target and not complete (like most of my personal pet projects) so don't expect enterprise level detail and a complete publicly consumable API.

The post after that will probably show you the product in action :-)

EOAST - Evolution of a software thingy - Part 1

Its been a little while since I have blogged as I haven't had very many value items to add to the blogosphere lately. However, I have been steadily (albeit slowly) working away on a little software project that may or may not see the light of day, but it has been a fun learning experience, and may eventually come to something. If nothing else, its a good testbed for a number of latest technologies.

I decided to chronicle some aspects of this software "thingy", hence the blog title "Evolution of a software thingy" - EOAST (you gotta have an acronym otherwise you don't exist :-) )

So this first post is the requirements if you will, the business drivers behind what and why I decided to do what I did.

Context

If you have read any of my previous rants, you'll note that I am not the biggest fan of the latest incarnation of outlook. I use it daily, and rely on it, but its slow, often locks up (particularly on slower links) and generally sucks the life from my system.

Additionally, like most people, I use a number of various tools to collect information for either storage, perusal, reference and a bunch of other things.

The collection of various information pieces shows a lot of similarity with each other, the main difference being collection frequency or scheduling, management of the information, and its presentation. For all the tools I use, none of them are exactly how I want them. Each one has different parts I like and dislike and merging the information they gather into a manageable unit is tedious, if not almost impossible in some cases. Think blog aggregators, email, twitter etc. Essentially, they all collect some information at different frequencies or schedules and display it for manipulation or searching, sometimes storing it for later retrieval or whatever.

That's the pain point part. I also wanted to have a purpose for some of my technological investigations. We all know that you can play with a new technology like Entity Framework, or Linq to SQL and make it do what the examples do. However, when hard pressed to do something that meets a requirement, not a technology demonstration, and suddenly the game changes, and efficiencies and/or deficiencies are far more apparent. So essentially I wanted to play with some new tech as well to get a real feel for its capabilities.

Requirements

So with that in mind, my requirements for my "Software Thingy" are as follows:

1. Must be able to collect information at various scheduling intervals, whether monthly, daily, hourly, every minute, or manually initiated. Additionally, it must also be able to push information, rather than simply request via a schedule. Think Microsoft exchange (push) and blog readers (pull).
2. it must store this information that it collects into a reliable store, SQL Server being the obvious candidate. This will cater for large volumes and easy searching. (Think problems with large Outlook PST files).
3. It must be reliable to handle badly written modules that it interfaces with. If something is taking too long, or just plain dead, the main application/engine will not die as well and render itself unusable (ummmm....Outlook again :-) ).
4. It must not be tied to one method of collecting information. It must be flexible enough to gather information from anywhere. Blogs, twitter, POP email, exchange email, manual entry into  some data capturing app, text files, anything.
5. it must not be tied to one display method. I want the ability to determine how particular information is displayed. I'd like to be able to map the gathering of information from one source, to a certain display method, or even multiple display methods, all administratively.
6. I don't want to do it all. I don't have that much time or smarts. I would like to make something very easy to develop display modules or information gathering modules so that I can get perhaps a WPF guru to make me a slick WPF display module, without having to care where the info comes from and how its collected. Similarly for the information gathering. I'll write the items that pertain to me, and allow others to easily bolt on a different form of information gathering and not necessarily care about how its stored nor displayed.
7. I want to utilise technologies such as WCF (my favourite), Linq to SQL, WPF and any other thing that makes sense.

Next up will be the initial design that I have come up with. Stay tuned.

MSFeedsSync - feeds Synchronization has stopped working issue

Recently (and quite randomly I might add), I had an error dialog popup continuously, every 5-10 minutes which read:

"MSFeeds Synchronization has stopped working (2054)"

It was driving me crazy and was unsure how to resolve. There is actually not that much on the net about but this post got me going in the right direction.

Basically, I first disabled the service by typing at the command prompt:

msfeedssync disable

Then I deleted all my files (which included many feeds I was no longer using) from the following dir:

\Users\%username%\AppData\Local\Microsoft\Feeds

then re-enabled the feeds service by typing:

msfeedssync enable

And all was good again.

VisualSVN - Subversion within Visual Studio

I am a big fan of Subversion. I like TFS too, but subversion is more suited to my small scale personal needs whereas TFS is more in the corporate space, or at least, more in the "requires more juice" space. Juice that I don't really have.

So, I use subversion for a majority of my personal development. I like it. I normally use TortoiseSVN for the explorer integration and have been playing quite happily with that.

Recently though, I grabbed me a copy of VisualSVN which adds Visual Studio integration for subversion. I like it, I like it a lot. It makes things very nice indeed. Here are some screen shots. Highly recommended.

Show Log

 

Visual Status Indicators in the solution

Context menu within Visual Studio solution explorer

ASP.NET Podcast Show #119 - Using the History (Back) Functionality with the ASP.NET AJAX Web Services in .NET 3.5 Service Pack 1 Beta 1

Wally has put out another podcast following his series around some of the latest AJAX features in .Net 3.5 SP1. In this installment, he looks at the history (back button) support.

Subscribe to Everything.

Original Url: http://aspnetpodcast.com/CS11/blogs/asp.net_podcast/archive/2008/07/01/asp-net-podcast-show-119-using-the-history-back-functionality-with-the-asp-net-ajax-web-services-in-net-3-5-service-pack-1-beta-1.aspx

Subscribe to WMV format.

Subscribe to M4V for iPod Users.

Subscribe to MP3 (Audio only).

Download WMV.

Download M4V for iPod Users.

Download MP3 (Audio only).

Download PPT associated file.

Show Notes:

• This show is an extension of Show #110 on ASP.NET AJAX and Live Maps.
• Back and forward are now enabled.
• Data stored on the url.
• Upcoming Community Events:
  • GiveCamp in Ann Arbor, MI. http://www.annarborgivecamp.org.  Check for Satellite events.
  • CodeStock in Knoxville on August 9.  http://www.codestock.org.
  • devLink in Nashville on August 22-23.  http://www.devlink.net.
  • INETA Silverlight Challenge..  http://www.ineta.org/silverlight.

Source Code:

     <asp:ScriptManager ID="sm" runat="server"
        EnableHistory="true"
        EnableSecureHistoryState="true" >
        <Services>
            <asp:ServiceReference Path="~/GetMaps.asmx" />
        </Services>
    </asp:ScriptManager>
<script language="javascript" type="text/javascript">
    var Map;
    var userNavigated = false;
    function pageLoad() {
        Sys.Application.add_navigate(onNavigate);
        GetMaps.MapData(SetupMap);
    }
    function onNavigate(sender, e) {
        if (userNavigated) {
            restorePage(e.get_state());
        }
    }
    function restorePage(PageState) {
        var UpperLeftLat = new Number(PageState.UpperLeftLat);
        var UpperLeftLon = new Number(PageState.UpperLeftLon);
        var BottomRightLat = new Number(PageState.LowerRightLat);
        var BottomRightLon = new Number(PageState.LowerRightLon);
        var ZoomLevel = new Number(PageState.ZoomLevel);
        var Lat = (UpperLeftLat + BottomRightLat ) / 2;
        var Lon = (UpperLeftLon + BottomRightLon) / 2;
        if (!(isNaN(Lat) || isNaN(Lon))) {
            Map.LoadMap(new VELatLong(Lat, Lon), ZoomLevel, 'h', false);
            GetMaps.GetPointData(10, UpperLeftLat, UpperLeftLon,
            BottomRightLat, BottomRightLon, GetDataSuccess);
        }
    }
    function SetupMap(result) {
        var Lat = result.Center.Lat;
        var Lon = result.Center.Lon;
        var ZoomLevel = result.ZoomLevel;
        var MapView, TopLeft, BottomRight;
        try {
            Map = new VEMap('MapDiv');
            Map.LoadMap(new VELatLong(Lat, Lon), ZoomLevel, 'h', false);
            MapView = Map.GetMapView();
            TopLeft = MapView.TopLeftLatLong;
            BottomRight = MapView.BottomRightLatLong;
            //TopleftLatLong and BottomRightLatLong return a VELatLong object.
            Map.AttachEvent("onchangeview", MapChangedView);
            SetupHistory();
            GetMaps.GetPointData(10, TopLeft.Latitude, TopLeft.Longitude,
                BottomRight.Latitude, BottomRight.Longitude, GetDataSuccess);
        }
        catch (e) {
            alert("Error: " + e);
        }
    }
    function GetDataSuccess(result) {
        var i = 0;
        var Lat, Lon;
        for (i = 0; i < result.length; i++) {
            Lat = result[i].Location.Lat;
            Lon = result[i].Location.Lon;
            var shape = new VEShape(VEShapeType.Pushpin, new VELatLong(Lat, Lon));
            shape.SetTitle("Title: " + i);
            shape.SetDescription(result[i].Description);
            Map.AddShape(shape);
        }
    }
    function SetupHistory() {
        var PageTitle;
        var now = new Date();
        var CurrentTime = now.getHours() + ":" + now.getMinutes() + ":" + now.getSeconds();
        PageTitle = "AJAX History Test Time:" + CurrentTime;
        MapView = Map.GetMapView();
        TopLeft = MapView.TopLeftLatLong;
        BottomRight = MapView.BottomRightLatLong;
        ZoomLevel = Map.GetZoomLevel();
        var PageState =
        {
            "UpperLeftLat": TopLeft.Latitude,
            "UpperLeftLon": TopLeft.Longitude,
            "LowerRightLat": BottomRight.Latitude,
            "LowerRightLon": BottomRight.Longitude,
            "ZoomLevel": ZoomLevel
        }
        userNavigated = false;
        Sys.Application.addHistoryPoint(PageState, PageTitle);
        userNavigated = true;
    }
    function MapChangedView(e) {
        Map.DeleteAllShapes();
        MapView = Map.GetMapView();
        TopLeft = MapView.TopLeftLatLong;
        BottomRight = MapView.BottomRightLatLong;
        SetupHistory();
        GetMaps.GetPointData(10, TopLeft.Latitude, TopLeft.Longitude,
        BottomRight.Latitude, BottomRight.Longitude, GetDataSuccess);
    }
</script>
    <div id="MapDiv" style="position:relative; width:450px; height: 350px;" ></div>

ASP.NET Podcast Show #118 - Peer-To-Peer with Windows Communication Foundation (WCF)

So I *finally* managed to get a videocast/podcast done. Wally has been harassing me for ages, and I eventually had the time to get one done. This one is on the Peer to Peer support in Windows Communication Foundation (WCF) and how easy it is to do. Its remarkably simple yet extremely powerful.

I hope you enjoy it.

Subscribe to EVERYTHING <-- What you REALLY WANT TO DO!

Original Url: http://aspnetpodcast.com/CS11/blogs/asp.net_podcast/archive/2008/06/19/asp-net-podcast-show-118-paul-on-peer-to-peer-with-windows-communication-foundation.aspx

Subscribe to WMV Video only.

Subscribe to M4V Video only.

Subscribe to MP3 Audio only.

Download WMV.

Download M4V.

Download MP3.

Show Notes:

• Understanding PNRP Clounds
  http://blogs.msdn.com/p2p/archive/2007/06/12/understanding-pnrp-clouds.aspx
• Peer Name Resolution Protocol
  http://technet.microsoft.com/en-us/library/bb726971.aspx
• Peer-to-Peer Programming with WCF and .NET Framework 3.5
  http://msdn.microsoft.com/en-us/library/cc297274.aspx
• INETA Community Champions Information.
• Silverlight Challenge.

Download Source Code

WCF services getting errors when hosting in IIS

This post is more for my own records than anything and I have been meaning to write this down for some time.

Problem:

Had a WCF Web Service that was being hosted in IIS. Worked in dev and test fine. Moved to prod and bang, I get the following error:

This collection already contains an address with scheme http. There can be at most one address per scheme in this collection.
Parameter name: item

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.ArgumentException: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.
Parameter name: item

 

Issue:

The production server had multiple host headers defined within IIS, meaning that the HTTP base address being passed through actually consisted of 2 base addresses with the HTTP scheme, which is not allowed when hosting WCF services in WCF.

The Solution:

Easy: Remove the multiple host headers. In my scenario, this was a viable solution as the multiple host headers were not actually necessary and represented some 'left over' configuration.

Harder, but more flexible and solid: Create a custom service factory to strip out the unwanted base address as outlined in a good post by Rob Zelt.

VSTS 2008, Web Tests and using Fiddler

I have been doing some performance and load testing of a web application I have been working on lately. Most of my previous experience was with the 2005 flavour of VSTS and I was keen to play with the 2008 version to see what improvements have been made. This web application is quite heavy in terms of Javascript and AJAX feature usage.

I am happy to report that things are much better in this version, but still not as easy as I had hoped. Form value extraction and recognition of AJAX requests seem to work well.

However, I still had some problems running the tests. For the most part they would run ok, but would cause errors on the server such as:

'Index was outside the bounds of the array.' at System.Web.Handlers.ScriptResourceHandler.ProcessRequest

which was always when accessing some script resource via the ScriptResource.axd handler.

I could see no reference to any of my code in there. Everything seemed to work as far as the client aspect was concerned (examining the results of the web tests) but this did not sit too well with me.

So I decided to try and record some tests via Fiddler2 HTTP proxy tool to see if that helped. There was definitely a little more work in getting this to record and playback web tests properly, but I was able to get some cleaner looking tests, and no errors generated on the server using this method. I did verify that all the functionality was being exercised on the server just in case I was missing anything and getting false positives.

The procedure I used to record web tests and get them working was.

• Load Fiddler2
• Load Internet Explorer and browse to the site, running through my specific performance test scenario.
• In Fiddler, press F12 to ensure no more sessions/traffic is captured.
• Select all the sessions by using CTRL-A (ensuing there was no other traffic that got recorded at the time. If there was, just highlight that session and delete it)
• Go to the File >> Save >> Sessions >> As Visual Studio Web Test menu option
• In Visual Studio 2008, within your test project, add in the existing web test.
• VS2008 will add some 'Form Field' extraction rules to each request.

• If you are using UpdatePanel's a lot and need to extract the ViewState from UpdatePanel delta responses, you will need to change the extraction rule.
• For this I created a Text Extraction rule.

• This rule simply looks for __VIEWSTATE| (yes, that's a pipe char on the end) as the start text and a pipe | as the end delimiter text and extracts everything in between.

• I had to add this extraction rule to each request, and replace the references to the old Form field extraction value

• with references to our new AJAX text extraction rule.

Like I said, it was a little bit of work to go through each page request and fix it in this manner, but the tests execute cleanly every time with no errors on the server as previously mentioned and it is actually a neater looking web test.

Profiling your AJAX Applications using AjaxView

Don't know if this has already been discussed at length before, but I found an interesting little tool from Microsoft Research for profiling the Javascript functionality in your AJAX applications.

Its called AjaxView and can remotely monitor any AJAX web app that is currently on the web, locally or whatever.

You basically, install a small proxy application, then change your Internet Explorer LAN connection to go through a proxy of localhost:8888.

Go and browse your web application or your own AJAX app, and execute your normal series of functionality.

Once your done, you then browse to a page generated by the tool itself

http://fakeurl.com/?&AJAXVIEWREQUEST=GET=main.html

This gets you something like this:

 

You can then click on whatever item you want, typically the site that represents the worst performance. You'll then get something like this:

Here I have clicked on the 'Mean Time (ms)' view to get column sorted by the slowest functions. If I want to drill down a bit more, I can click on the 'Details' link which gives me:

This shows me a breakdown of which functions took how long.

Its actually a pretty quick way to get a good idea of what is consuming a lot of client side cycles from a Javascript perspective. The tool is a little raw and is still somewhat rough, but its easy enough to use and get going and worth a quick look.

Visual Studio 2008 Team Test Issues

I have had ongoing problems getting various tests to work using my installation of VS2008 Team Suite. Initially, I had a problem running unit tests (see here), then sometime after that, I had other niggling issues with code coverage and had to copy assemblies from Private folders to public folders and vice versa.

Recently, I tried recording some web tests so I can perform loading testing on a particular web application. As luck would have it, it was failing with the cryptic message shown in below:

Method ‘CreateDataSourceWizard’ in type ‘Microsoft.VisualStudio.TestTools.LoadTest.WebTestEditorPane’ from assembly ‘Microsoft.VisualStudio.QualityTools.LoadTestPackage, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ does not have an implementation.

Google gave me no juice on this and frustration was setting in.

I tried the following:

• Lots of time checking versions of files and reflecting back over the assemblies mentioned in the error message.
• Un-installed and re-installed the Team Test components.
• Un-installed and re-installed Visual Studio 2008

Nothing worked. Oddly, when I un-installed the Team Test components, I still had 'Test' options functioning in my VS2008 menu.

So I finally resolved this issue, and the solution was this:

1. Un-install Visual Studio 2008
2. Go to the directory where it was previously installed and manually remove the copious amounts of files and directories that were left over.
3. Perform a registry clean using this Registry Cleaner for Vista.
4. Re-installed Visual Studio 2008

And now it works. Finally.

New feeds for the ASP.NET Podcast

Wally has created some new feeds for the ASP.NET Podcast  There is still the same general feed.  It will have all shows and all of the variations of the shows in it.  These include WMV, MP4/M4V, MP3, and whatever else one can throw in it. The rest of the feeds are:

• WMV feed.
• MP4/M4V feed.
• MP3 feed.

If you have any problems, let me know ASAP!

Latest Podcasts

Wally has put up some new podcasts around using Google maps and virtual earth with ASP.NET AJAX.

You can check em out at the following links:

Integrating ASP.NET AJAX with Google Maps.

ASP.NET AJAX with Virtual Earth

These podcasts allow you to explore mapping possibilities with the 2 popular mapping services currently available.  in combination with ASP.NET AJAX.

Check em out and get geographical !

WiX - Painful at best

So I finally got WiX to create an installer for me after many hours of poor documentation, obscure errors and bizarre angle bracket syntax.

The latest error I have really annoyed me as there was no obvious indication what the error was and how to resolve it, which is really the purpose of this post. To share this so that others may benefit, and because my memory is so poor, I need to record it for later use :-)

Basically I got this:

candle.exe : error CNDL0001: Cannot set column 'Attributes' with value 239 because it is greater than the maximum allowed value for this column, 127.

Nice. I wont bore you with the vulgarities that I produced trying to decipher this message, but it ended up being the fact that I had put a <sql:sqlDatabase> server definition was inside a feature element rather than outside on its own.

So I had put this:

<sql:sqlDatabase Id="SqlDatabase" Server="[DATABASESERVERNAME]" Database="[DATABASENAME]" />

inside a <Component> tag.

Placing the <sql:sqlDatabase> tag back outside the <Component> tag fixed this issue.

Hope it saves others similar frustration.

Remix in Oz

Remix, is heading downunder and registrations are now open. You can register here.This event is especially cool for the many people who did not get a chance to attend the actual Mix event in the United States (which is many of us).

Dates and locations are:

Sydney

   May 20
   Powerhouse Museum
   Harris Street, Ultimo

Melbourne

   May 22
   Melbourne Town Hall
   Cnr Swanston & Collins
   Street, Melbourne

There will be heaps of quality technical content from a bunch of local technical rockstars, and I will be attending and participating as an expert in one of the "Ask the Experts" areas.

Look forward to seeing you there!

Oh, by the way, each attendee receives a copy of Expression Studio 2 (rrp$1040) for free when they register!

MCTS in WCF

So a little while ago, I mentioned that I did a WCF beta exam for a Microsoft Certified Technical Specialist. Its been a long time since I did that and I had kinda forgotten about it, but today I received the good news that I am now a MCTS in WCF!

WooHoo!

I really thought I had bombed that exam due to the very little prep that I had done, but fortunately that is not the case.

I would love to hear from anybody else (particularly any fellow australians) who have also done the beta exam and received their congratulatory email.