Anyone using Urchin has probably noticed that your about 24-36 hours delayed (by my estimates). If I check Tuesday I can view Sundays stats. Now at some point Tuesday I can see Monday but then on Wednesday the information for Monday has changed (increased). Whats with the delay?

Prior to I was running a dreaded Yahoo! store with a basic free Site Meter and I was getting 'Real Time' stats. I could even see how many people were currently on my site and see where they were going, what products they were viewing, how many pages they've visited, how long they stayed, etc...

Is there anything in the works to get the stats up to realtime and also allow you to view atleast the top pages for products?

This would easily be one of the greatest improvements Monstercommerce could implement. (besides changing title tags, and deeper category levels)

Yes! I noticed the delays also! Its very important for us to have up to date stats to determine our business strategies for the week. Its kind of a bummer that hasn't hooked up with a good stats company yet. With all their programmers you'd think they could have their own stats program.
I loved the previous stat company because it showed who was on the site at that time and what they were doing/ how long they were staying. But, I also loved seeing the world map and how many visitors from what areas.
I know that the previous company had problems with staying online, but they provided the best stats!
Hey Captain, could you pass these comments on to your management? thanks, its important to everyone
Anyone who reads this and feels the same, please post a yes so that knows that people do need good up to date stats

I agree. The previous Stats program was so much better.

www.mineralbeauty.com

Thanks,

Glad to know I'm not alone in my thoughts about this.

I can check into the delay issue... however I disagree that you would want to go back to deep matrix.

Urchin is the most widely used stats program on the net and it is a superior product. They are the industry leader. If you require an alternative stats program you may want to consider web trends. I have used them and Urchin for a long time...

I think you can get webtrends for 99/month (that's what I currently pay for one of my sites.

I will keep you posted.

PS> We changed from the past stats program becuase it was unreliable.... and grossly inaccurate.

I don't know anything about Deep Matrix, but I'll totally take your word on that one Captain. We would definately want a stable system.

Let us know what you find out about why it is delayed and if your aware of any future plans for stats.

You the man Cap!

Thx

Okay as for Stats. I have 3 stores and two do pull stats in about 24 hrs. The Main 1st Store has not posted since Tuesday up till 6:00 pm. It is Friday. I do not consider this to be reliable info. I do miss the real time graph of the other Stats system and the geographic info. I do like that urchin seems to filter out the aol proxys and other that are not real visitors. There is always good and bad with everything but I would rather have the old system at this point.

I have spoken with Andy... Director of tech support regarding stats. He is looking into the issue. I will post as soon as I hear an answer. Delayed stats are useless! We'll get this squared away!

Much appreciated Captain! Thanks.

gdogsd13,

What is the site that is not working properly with stats? If you want, you can send me a personal message, or you can just post it. If it does not fully update within 24 hours, there is a problem. I will get this problem fixed for you on your site that it is not working properly. As for the real time stats, Urchin is not able to do that at this time. They do have a lot of nice features about them, but that is not one of them. I am currently working on a problem similar to this, and it might even be you.

I have seen two threads on the forums thus far dealing with stats, and since there is a lot of general information to cover I will be posting the same response and clarification to both threads, as I feel a lot of the information is beneficial to both topics.

Prior to my joining the MonsterCommerce team some time back, we used a program called DeepMetrix LiveStats for web analytics. This program had a very nice set of visual displays, but had many horrible stability issues. As I am sure some of our veteran customers recall, LiveStats would go down almost predictably every other week and typically required us to get the actual development team from DeepMetrix working on correcting various issues with the database for stats, or replacing/patching the actual binaries almost every time. This led to issues where the stats service would be down for days while waiting on repairs, inconsistent reporting, etc, etc.

During this time, I was charged with evaluating other web stats solutions, and out of the large group of options a technical group decision was reached on Urchin. Urchin has proved to be a fairly robust solution, we have no had any crashes per se, and performance is fairly reliable...with a few minor issues/points of concern, which I will address further in this post.

One of the primary issues is that customers want to have access to tracking of variables in their web site URLs such as http://www.monstercommercedemo.com/product.asp?3=146, so that an accurate gauge of how popular some products are, or the progression of browsing from the customer, etc, etc. Unfortunately, this is just not something that our standard stats offering will be able o provide for several reasons. The first of these reasons being that the added overhead processing the URI stems for parameters of page requests causes to the overall performance of the stats server is so high that it literally adds almost exponentially to the total processing time of all sites. This ability does exist, and it has actually led to some serious delays in stats processing, which in turn makes other customers unhappy (especially those who don't seem to need more information than is already present). We are currently in the process of removing these filters to parse URI stems for the sites we enabled it on to improve overall performance. The second main reason we do not process log information to this granular a level is that configuring the filters of regular expressions it takes to catch and isolate the particular parameter or variable that our customers want is a manual process and would have to be continually modified and/or tweaked to catch exactly what information each customer wanted. Aside from the amount of time and effort this takes, it is literally something we would have to update or change every time a customer's product line was modified in any way.

MonsterCommerce charges a fairly nominal fee for a basic stats service.
With this basic service, a very nice amount of information regarding page views, referrers, general traffic levels and bandwidth usage, and time based analysis for traffic patterns is provided with our basic stats offering. Currently we do not provide geographic data information as a default option, but when the next major release of Urchin is made available (version 5), the processing time is expected to improve so much that we should be able to enable this feature. Again though, this really comes down to a "get what you pay for" scenario....for approximately $10.00 per month you really can't expect the equivalent of your own dedicated private stats server with full customizable reporting and display options, filters, etc.

What I can tell everyone is that there are several workaround options.
We rely on log file data for processing all of our sites bandwidth usage and have just reached a 90% completion point on analyzing a fully accurate monthly analysis of our customer's disk and bandwidth usage compared to their plan type. This is relevant to the stats discussion in that now we are able to isolate customers who are exceeding their standard plan types, or even the tiered options, and require them to upgrade accordingly. The biggest help is that many customers who are exceeding the usage for their tiered options are basically being told they have to go with dedicated servers, which at the same time will be running their own personal Urchin installation to analyze just the stats for their sites on their dedicated machine.

This does a couple of things....most apparent is that the load on the servers of all our plan types is decreased to optimal performance levels based on the plan type/allocation we have designed and implemented.
Additionally, these sites pushing massive traffic typically have gigantic log files (anywhere from 1 to 2 gigabytes in size *daily*). By removing the overhead of processing these extremely large log files (compared to our typical customer's log files that it), the processing time for Urchin on the shared stats servers is boosted. A less apparent benefit of this course of action is that the customers with dedicated servers have full control over how their log files are processed, including parameters/variables, and whatever kind of filters they want to apply....allowing for the more specific types of reporting some customers are asking for.

As I mentioned earlier, this is a relatively new plan of action (as of this month we have implemented). This means overall improvements to the stats performance as regarding update generation times should be seen from this month on. Also, a new group of stats servers is being brought online to make sure overall log analysis efforts are spread amongst many machines (basically you will not have to worry about this so long as you go to the Site Statistics link in your MC admin interface to get to your stats login). One of the best outcomes of the option to have a stats service running on a customer's dedicated store server is the level of tuning and control this provides....so I would like a lot of the customers who want more frequent updates and more granular reporting to keep this option in mind. The additional charge for stats service on your dedicated server (should you choose to go this route) is $35.00 monthly. If you want to see what a deal this is then checkout Urchin.com and you will see that the same one time charge for a purchase of a dedicated Urchin setup is about $500.00.

Moving on, other options are that we can make your log file directory available to you via a virtual directory (i.e. http://www.site.com/logs) and you (the customer) can feel free to download the logs and process them on your end in any manner you would like. The only caveats to this option are that the log file directory will only be available to you in read only format (with no exceptions as we need those logs for tracking bandwidth and disk usage), we do delete each previous months logs at the beginning of each current month (i.e. the first week of July all logs for June will be deleted and are not recoverable), and most importantly the downloading of your log files *will* count towards your total bandwidth usage figures. If you would like to have the accessible log file option enabled for your account, please submit a ticket specifically requesting this and we will take care of it. There is no charge for this option and it does not preclude being able to still get stats service with Urchin.

While we are on the track of processing your own files...please keep in mind that if you have personally found a really good solution that seems to be better than our current implementation, please do not hesitate to let us know! MonsterCommerce as a company knows that our relationship with our customers is a partnership, and your feedback/suggestions are taken very seriously....plus I can honestly tell your our technical staff has not had a chance to evaluate every option out there and we would appreciate the advice of anyone willing to share it.

This brings us to the final option for stats....and that is to go with a 3rd party solution such as WebTrends Live/WebTrends Reporting Service where snippets of code can be placed into your store pages and information sent for processing based on this to WebTrends, where reports are generated. I have spoken with WebTrends technical and sales staff and I can tell you honestly if WebTrends cannot track whatever degree of information about your web site you want...then it just isn't trackable. Keep in mind as I mentioned about price earlier, should you choose to go with an outside service such as WebTrends, you will be dealing with them, not us, and while we will do everything we can to integrate whatever code is needed for implementation; whatever technical support or billing issues you will have would be through this third party. Also, plain and simple...for the fine tuned information levels some people are mentioning they want, you will be paying quite a lot more. Basic reporting via WebTrends starts at $35.00 per month for 50,000 page views processed, with an additional $0.65 per each additional 1,000 pageviews processed. To put this in perspective, for some of our sites with 100,000 page views a month, WebTrends service would cost about $67.00 per month, and this cost would only grow as your traffic did (for one of our busier sites with 1,000,000 average page views per *day* the WebTrends processing service would cost $18,560 per month - these are actual quotes I received when investigating this option). Again I cannot stress enough, you get what you pay for is what a lot of this boils down to.

So, in summary the stats service we provide is basic. Stats are collected daily, with the previous days reports being available to our customer each day, a service we charge $10.00 per month for. The stats service will continue to be basic in regards to information provided and frequency of updates because of the pricepoint is set as a shared basic option (so just general information as opposed to very detailed reports). There are 3 courses of action if this is not acceptable to you....these being getting a dedicated server with your own stand alone urchin installation providing more frequent updates and more control over reporting information, processing the logs on your own end with the log files for your site being made publicly available via a virtual directory link (must request this option via ticketing system), or going with an outside solution whose implementation will not impact the overall server performance (i.e. we will not install the processing server on the same machine as the shared sites, code inside of your web pages is preferred). Finally, we (the technical staff) are aware of some performance problems and improvements needed and are focusing on these efforts as a priority, with noticeable results expect to be seen inside of the next month and then again once the next release of Urchin is available.

Wow, I wrote a book...sorry for the length, but I did want to cover all issues related to stats and basically inform you (our customers) of what is currently going on since typically the technical operation staff are kind of behind the scenes. What I can tell everyone who has managed to suffer through the entire length of this post is that my team is just as concerned about the satisfaction levels of our customers as any other department or person in MonsterCommerce is and that I can assure you our goals are to work towards continual improvement and to constantly do our part to maintain or increase our customer satisfaction levels.

Adam McCarthy
Senior Systems and Security Administrator
MonsterCommerce
adam@monstercommerce.com

Hey Captain,
I can understand the stability issue and that is why you switched to the new company. But, you are kind of in the same boat, you are having stability issues again.
Why couldn't DeepMetrix work the stability issues out? If they weren't willing to, than I say you made a good decision switching.
But, what I was pointing out was that DeepMetrix offered better stats. If they were stable than it would have been awesome.
Another question, you keep saying that WebTrends is the best stats available, than why can't they be the Stats provider?
Please don't take this as being mad or complaining (I love ), but I hope that a stable stats provider can be found that provides the type of stats DeepMetrix provided.
Thanks

Good post Strapworks, I completely agree with you. I'm currently using a third-party company to supply my stats for $7 a month so that I'm able to get realtime stats and more detailed information. Being a small business I'd rather not spend that $84 a year (all the small expenses add up) but I feel I don't really have a choice right now.

StrapWorks,

I don't take your posts as negative... it's healthy conversation. The reason we don't use WebTrends for clients is because it is too expensive for our clients.

I use webtrends for my sites and MC uses it for their sites... but the majority of clients can't afford it. MC is limited with stats options because of the cost...

If you can afford it... I recommend WebTrends.

I will look into DeepMatrix issues again with Adam. Did you read his long post?

LOL, I am not much of a book reader, but I hit most of the points and I understand that it is just a basic service that you guys are trying to make better, thats what I like about you guys, you take customer feedback and make your service better
I'll check out webtrends
Hey UltimateKeyChains, who do you go with?
Later Alligator

Have we solved anything with the Stats issue discussed above. I am just curious since my store is now 48 hours behind with Urchin

Urchin still is having its problems. I just went in today and it took me to the week of 7/27- 8/2 and said I had some hits! LOL, I just hope Urchin has an idea of how to fix the current problems and is willing to fix them quickly

Yeah, I still have random days over the past month that show "0" hits. I find this hard to believe since "I" hit my own site atleast 5 times a day from 2 different locations. Since I'm not the first one to mention still having the problems I assume that all the changes mentioned previously in that long post didn't resolve the issues.

I am having the servers admins check for you right now...

Hello Ultimate and Strapworks-


Yes, Urchin is not fool-proof software. For some reason, it skipped 07/08 for you, Ultimate. I have now re-run Urchin for that day, and they are now there.

Your issue, Strapworks, is inherint in Urchin. Sometimes, if a person browsing your site sends you a malformed request, Urchin has difficulty understanding what it is, and may put it at a date that hasn't happened yet. What compounds the problem is that Urchin defaults to the most recent date to view stats (which makes sense under normal circumstances) but then it shows you the random data that is actually incorrect. We are told by them that this will be corrected in the next version of Urchin.

Thanks Sabres
I do hope that Urchin takes these problems to heart, because it can be easily fixed (at least I think) and than they would be almost problem free.
I just wish they had some fancier stats, just my little dream.

WHat i haven't been able to track in Urchin are specific links. I sent a newsletter out with links like this http://www.etoolcart.com?source=tailgate so that I can identify when a click happened from the newsletter. The link works properly but urchin doesn't pick it up. I subscribe to a tracking site but really hate to put links like http://www.linkcounter.com/go.php?linkid=274210 especially in the text version of my newsletter . I think it is tacky and uncomfortable for the recipient to know we are tracking him.

Anyone have any ideas on how I can write a link Urchin can track?

I have a couple of questions brought up by the large post. I also want to make clear that I have been very happy with MC in general and this post is not intended to bash anyone person or group within MC.

1. If the detailed log files are being pushed to a DW for Urchin to read, why are we experiencing the performance delays at all?

2. My stats are somewhere between 24 and 48 hours off; however, I can live with non real-time information if the gap narrows. Do you expect to be able to narrow the gap to within 24 hours in the next month? Please advise.

I would be happy if we were able to access our logs until they availiable from urchin.

LHunter,

Let me check with Adam.

We'll let you know shortly.

Its wed. the 30th and my newest stats are from sunday the 27th.

also, Thu 7/24 shows i had 0 visitors (which is impossible).
Just curious as to what is going on. The stats seem to be getting worse!

I am just curious if there is a stats program out there that doesn't have problems? If there are than why does always seem to pick the winners? LOL, just a joke
I guess I can answer my own question. Because the best stats programs charge the highest prices, and we can't possibly pass those costs back to our customers.

StrapWorks...

You hit the nail on the head... I recommend WebTrendsLive.com for stats.

This package (depending on the options you choose) will cost more per month than your storefront costs... but if you have a successful business you cannot afford not to have a robust stats package.

The stats provided for $10/month through MC is for users that want a cheap solution to web analytics. If stats are very important to you... you should not be running the $10/month package. It will not be adequate.

I do not use MC stats for my stores. And MC certainly doesn't use MC stats for MC's sites. We pay thousands a month for stats.

We looked into providing more robust packages... but the cost to license per user woul have cost more than the monthly fee for the storefront... so it didn't make sense. We had to choose a solution that was affordable for our smaller clients. The larger clients use premium programs that costs much more.

Summary >> MC stats for $10/ month are going to be limited if you require robust analytics. If you need more, I recommend implementing a more robust solution like www.webtrendslive.com. Just be prepared to pay more!

Hope this helps...

Regarding issues... please refer to Adam's response:

Two things -

1) Long story made short - updates are done daily, with the previous days log files being processed. This is done because it is the most consistent, reliable, and stable way to update stats for all users in a shared resource environment.

Also, dedicated server customers get their own Urchin installations, and we can and do process/update stats hourly for these customers.

2) A lot of the previous concerns or issues with Urchin 4 should now be resolved with Urchin 5 and from everything I have seen (and from customer feedback), this does in fact seem to be the case.

I just checked my Urchin Stats and show zero for Sunday. Anyone else have this issue on Sunday or was it an issue with our site

Yeah me too, probably due to conversion to v4.0..

Actually, my stats are almost non-existant since my 4.0 migration. Not sure if it's a Urchin thing or something caused by the migration...

I to wanted some realtime stats and I have been trying out hitslink stats at hitslink.com. This is a very good stats package in my opinion and I am going to purchase once the trial runs out. may want to think about this for an option. You just past the code on your homepage. Its real stats with very good detail and with the enterprise edition you can track ppc campaigns as well. Under 10,000 hits a day is roughly 20/month then for 30/month I believe you can take it up to 20,000. A bargain if you ask me.

Urchin Stats is good, it is just delaying because of the v4.0 conversion. I would not want to dish out another monthly fee for stats... ....monthly fees can make business expenses high, so whenever we get a chance, we like to pay a one time fee only...

Ultimate,

I have forwarded this post to Admin Guy for review... he'll reply to you shortly. Shouldn't be a difficult fix.

Urchin stats not working post-migration are usually caused by either not updating the new location of log file information in the stats server properly or at all. If you continue to encounter this issue or have it occur after your migration, please submit a support ticket or call support and we will get you handled ASAP.