SUMo scanning resp. detection algorithm

scheff
Posts: 92
Joined: Tue Apr 16, 2019 3:00 pm
Location: DE

SUMo scanning resp. detection algorithm

Post by scheff » Mon May 06, 2019 3:45 pm

Can you please provide further details how SUMo works for detecting installed software applications?

For example, I'm running SUMo under Windows 10 operating system. And it looks like SUMo doesn't find any single Windows 10 App although Windows reports me between 10 and 60 installed, depending how I query which tool. Am I right?
How do I need to configure SUMo to find installed Windows 10 Apps?
How do I need to configure SUMo to find nearly all installed Windows 10 Apps including Windows 10 system Apps which were preinstalled?

I already tried if it helps to enable Microsoft products being reported as most installed Windows 10 Apps are of Microsoft. But this didn't seem to help neither or only a little bit.

I tried to find an article about standard storage locations for installed software on Windows 10 but couldn't find an exhaustive one before aborting this search. I found one storage location. But there seem to be installed just a few apps, not the majority of apps.

scheff
Posts: 92
Joined: Tue Apr 16, 2019 3:00 pm
Location: DE

Re: SUMo scanning resp. detection algorithm

Post by scheff » Mon May 06, 2019 6:22 pm

After experimenting a bit with the Microsoft Product enabled configuration, the next question raises on the algorithm. (It provides a hint to answering my question on user base.)

As a feature update of Windows 10 is already released and pending start of rollout, expected by the end of this month, it is a good idea to select a component of Windows to be updated as reported by SUMo and which is present in every Windows operating system in order to look up on SUMo server how many SUMo users have this component installed and how many in which version in order to get an idea of the SUMo user base. So I expect about the same number of users having such a component installed as SUMo users exist which is more than 2000. But instead I get numbers often below 10 SUMo users! Why?

With this look up on SUMo server, I change the query to get an overview of all Microsoft Products by any SUMo user (https://www.kcsoftwares.com/sumo/compan ... =Microsoft ). It gives a good overview and raises further questions. IT reports more than 450 products of Microsoft installed by any SUMo user. Taking a closer look, there is obviously the distinction between 32 and 64 bit systems. But I discover further that localization of these components are counted as distinct products although they should have the same executable. I don't remember when Microsoft stopped shipping different binaries for different localizations. I think they still had this distinction with Windows XP and Windows Vista but think they already dropped it with Windows 7. Can anybody confirm or otherwise correct?
And why does SUMo server then still count these identical binaries as different although it's just their localization and hence presentation, not the version which differ?

But even if I make a summary of those localizations and bit editions, I'll come to SUMo user numbers in the order of roughly about 50 not 2000. Why?
This let's me assume a bug in SUMo for a wrong handling of SUMo configuration option. Is it possible that SUMo not only suppresses reporting Microsoft products to its user by default unless configured not to suppress which would all be the expected behaviour, but instead even suppresses sending its detection of Microsoft products to SUMo server?
If this assumption is true, then the numbers found on SUMo server should correspond to the subset of SUMo users only who have Microsoft Product reporting enabled in SUMo. That would explain the above numbers, taking into account the warning in the settings menu not to enable reporting.

Kyle_Katarn
Site Admin
Posts: 1298
Joined: Sun Jul 03, 2011 8:13 pm

Re: SUMo scanning resp. detection algorithm

Post by Kyle_Katarn » Mon May 06, 2019 7:54 pm

SUMo detection is based on scanning shortcuts (desktop, start menu,...) + a few exceptions (based on registry scanning).
When Microsoft support is disabled, they are no longer listed (and therefore no longer reported to the server for update version computation).
As a consequence, support of Microsoft products in SUMo is disabled by default and not recommended.

scheff
Posts: 92
Joined: Tue Apr 16, 2019 3:00 pm
Location: DE

Re: SUMo scanning resp. detection algorithm

Post by scheff » Tue May 07, 2019 9:12 am

Kyle_Katarn wrote:
Mon May 06, 2019 7:54 pm
SUMo detection is based on scanning shortcuts (desktop, start menu,...) + a few exceptions (based on registry scanning).
When Microsoft support is disabled, they are no longer listed (and therefore no longer reported to the server for update version computation).
As a consequence, support of Microsoft products in SUMo is disabled by default and not recommended.
Hi Kyle,

your answer doesn't seem correct nor sufficient.

I don't like my desktop full of shortcuts. Hence I disable creation of shortcuts on software installation as far as installers provide this option. If an installer doesn't provide such an option and nevertheless creates a desktop shortcut, then I check if I find a corresponding entry in the start menu. If yes, I delete almost all those desktop shortcuts. Although I thought that one desktop shortcut has been ignored, I found that it is not. Also renaming of desktop shortcut doesn't lead to ignoring nor false reporting. It reports the original name provided by the software publisher and found in the start menu. The few exceptions found in start menu and on desktop are not counted twice as it is one device and one identical installation, not a side-by-side installation. So for this small part, SUMo works as you claim.

But what do you mean by start menu. Do you really mean the start menu of Windows?

If yes, then your claim is not correct as already written above, i.e. on Windows 10 Apps which are in the start menu.

In Windows, the start menu presented by the operating system to the user is a federation of several source providers. With Windows 8, Microsoft split classical start menu for classical programs and new start interface for new (often touch pad/screen based UI) applications. With Windows 10, both start menus have been merged and presentation became more configurable. Before Windows 10 one source of the classical start menu was the file system section of the particular user as still is the case, another source was the default user section in the file system which I don't see anymore in Windows 10 although its predecessors. And still other sources were configured in the operating system with the user able to change this standard configuration (start menu configuration) in all versions of windows for decades already. With the merge of professional Windows operating systems and consumer Windows operating systems to a common Windows operating system, the handling of the first installed user on installation still had another source called main user section unless already on installation the computer is already member of a Windows domain based network. With Windows 10 system Windows 10 applications, preinstalled Windows 10 applications and user installed Windows 10 applications are also found in Windows 10 start menu. As long as Microsoft Product reporting is not enabled in SUMo, SUMo didn't find any single Windows 10 application although found in the start menu. And if this reporting is enabled, it only finds very few, still ignoring most others. I don't know how to query which Windows 10 tool to find all these apps. It seems to me that Windows provides various means resulting in different subsets reported while SUMo ignores them all by default. That's why I asked, how I have to configure SUMo so that it finds them and still couldn't read an answer, neither in your preceeding reply post nor in any other although I read many not all depending on age and subject.

And as I already wrote, the default configuration not to report Microsoft products in SUMo does mean that they are not listed in the report of installed software nor on available updates. It does not mean that they're not detected nor scanned nor suppressed in reporting to the SUMo database server in order that SUMo users having enabled this SUMo option get reliable reports. So it is a bug that this kind of software is not reported to SUMo server in the default configuration.

As I wrote already, Microsoft, Windows and a few other software use localization very deeply and intensively replacing the real names of applications and storage path by virtual localized ones. If I query Windows store for installed Windows 10 apps, I get their real name as installed. If I query Windows control panel, I get them reported by their localized name. The same applies for other kind of software. SUMo doesn't take this into account and reports those localized names instead of the real names on SUMo server and hence probably also on SUMo client. Why?

Kyle_Katarn
Site Admin
Posts: 1298
Joined: Sun Jul 03, 2011 8:13 pm

Re: SUMo scanning resp. detection algorithm

Post by Kyle_Katarn » Tue May 07, 2019 7:03 pm

Have you looked at SUMo logs in debug mode (lines added before & during the scan) ? This indicate where SUMo looks for links.
Should SUMo be missing a key source of information, please let me know :)

scheff
Posts: 92
Joined: Tue Apr 16, 2019 3:00 pm
Location: DE

Re: SUMo scanning resp. detection algorithm

Post by scheff » Wed May 08, 2019 2:30 pm

Kyle_Katarn wrote:
Tue May 07, 2019 7:03 pm
Have you looked at SUMo logs in debug mode (lines added before & during the scan) ? This indicate where SUMo looks for links.
No. It doesn't due to some hidden limits. When I enable debugging mode, the log file gets so large that something cuts it so that the complete phase before scanning and the main part of scanning is already lost before being able to look at the log file! When I'm able to view the log file via corresponding SUMo menu entry, it starts the default editor of Windows to open the log file with 10000 lines. All lines older than these 10000 lines are not in the log file displayed by the Windows default editor. Is this a (probably hard coded) limitation of SUMo logging or of the Windows default editor in which case it would be sufficient to define another editor as default editor?
Kyle_Katarn wrote:
Tue May 07, 2019 7:03 pm
Should SUMo be missing a key source of information, please let me know :)
Complexity of Windows has grown in the past decades so that my knowledge is too limited (and outdated) for knowing and supplying such information. I neither don't have a corresponding MSDN subscription to see if such information may be found via this mean.

scheff
Posts: 92
Joined: Tue Apr 16, 2019 3:00 pm
Location: DE

Re: SUMo scanning resp. detection algorithm

Post by scheff » Wed May 08, 2019 2:34 pm

scheff wrote:
Wed May 08, 2019 2:30 pm
Kyle_Katarn wrote:
Tue May 07, 2019 7:03 pm
Have you looked at SUMo logs in debug mode (lines added before & during the scan) ? This indicate where SUMo looks for links.
No. It doesn't due to some hidden limits. When I enable debugging mode, the log file gets so large that something cuts it so that the complete phase before scanning and the main part of scanning is already lost before being able to look at the log file! When I'm able to view the log file via corresponding SUMo menu entry, it starts the default editor of Windows to open the log file with 10000 lines. All lines older than these 10000 lines are not in the log file displayed by the Windows default editor. Is this a (probably hard coded) limitation of SUMo logging or of the Windows default editor in which case it would be sufficient to define another editor as default editor?
I forgot to ask the resulting question:
How to configure SUMo in case of debug enabled to apply much larger logging limits compared to non-debugging mode?

It doesn't seem possible to configure it via GUI. Which other means are implemented for such configurations at least for my installed version of SUMo Pro?

Kyle_Katarn
Site Admin
Posts: 1298
Joined: Sun Jul 03, 2011 8:13 pm

Re: SUMo scanning resp. detection algorithm

Post by Kyle_Katarn » Wed May 08, 2019 3:15 pm

I've never reached any limit on log file. However, you're probably right, there must be some OS-related limits (unlikely due to filesystem anyway)
In you case, i'd suggest :
- restart SUMo
- run a scan
- open log

Note : Please do not try to open log from Explorer while SUMo is running since, if you're not in debug mode, log is saved to file system only when closing or using ?>Open log (which may be the cause for your missing data).
Note2 : when in debug mode, log is saved on file system every time a new item is added which significantly affects performance (eg : please do not use debug mode when not needed)

scheff
Posts: 92
Joined: Tue Apr 16, 2019 3:00 pm
Location: DE

Re: SUMo scanning resp. detection algorithm

Post by scheff » Wed May 08, 2019 4:42 pm

Kyle_Katarn wrote:
Wed May 08, 2019 3:15 pm
I've never reached any limit on log file. However, you're probably right, there must be some OS-related limits (unlikely due to filesystem anyway)
If this part of my knowledge is not outdated, the limit observed and reported by myself is far less than the OS related limits. I'm running Windows 10 with its default file system as far as I know. As my system disk is a SSD attached via PCIe, I think the firmware for SSD controller will do some different mapping and emulation to the higher driver levels of the file system.

When opening the log file via SUMo menu, it starts the Windows default text editor. I can imagine that the limitation comes via this editor, not the file system nor the operating system. Even Windows comes with further editors, and I've installed at least one other. And Windows allows to change configuration for standard text editor. So this is worth a try.

And if this is not yet sufficient, it might be conveniant to further disable some automatic tasks SUMo performs on its start up. (I haven't SUMo configured to be started on the start of the operating system, if I understand the other configuration option correct.)

I was asking this question on limits due to my observation and not knowing if its coming from SUMo design. And would not be uncommon that some kind of ring buffer implementation would be used for handling limited resources in debug mode. BTW, the observed limit of 10000 lines corresponds to almost 1 MByte of logfile.
Kyle_Katarn wrote:
Wed May 08, 2019 3:15 pm
In you case, i'd suggest :
- restart SUMo
- run a scan
- open log
I'll give it a try. But before I want to follow my ideas mentioned above in order to see if the assumption is right that the default standard text editor of Windows is the culpit one. If what I'll find then isn't self-explaining (with my technical background), will you need the whole log file or which interesting extract otherwise?
Kyle_Katarn wrote:
Wed May 08, 2019 3:15 pm
Note : Please do not try to open log from Explorer while SUMo is running since, if you're not in debug mode, log is saved to file system only when closing or using ?>Open log (which may be the cause for your missing data).
As you asked me to enable debugging for this purpose, this note is irrelevant, just a reminder of what you replied in another post (https://www.kcsoftwares.com/forum/viewt ... 2914#p5123) of THIS threat above. We're writing here of debug mode logging since your just preceeding post of this threat. So even opening the log file becomes possible out of Windows Explorer. As mentioned in my initial post of THIS threat, I didn't find a log file in the file system with still running SUMo client in standard mode.
Kyle_Katarn wrote:
Wed May 08, 2019 3:15 pm
Note2 : when in debug mode, log is saved on file system every time a new item is added which significantly affects performance (eg : please do not use debug mode when not needed)
That's obvious. I've been a software developer in the past and still support software developers and their project leaders when in job. But when I started as developer, it was in a multiprocessing operating system context even if the development platform wasn't multi-process and I offered hints and hands-on training to colleagues to debug in such a mixed context by classical tools. But as race conditions are important for analysing and testing in such a context, alternatives are required to file base logging. Did I understand you right that you did not yet implement such alternatives or which other alternatives have you already in use for another kind of debug mode with less performance impact?
I can imagine that even your development environment comes with such an infrastructure at software level which should be sufficient for actual development workstations with multi-processor CPUs.

Kyle_Katarn
Site Admin
Posts: 1298
Joined: Sun Jul 03, 2011 8:13 pm

Re: SUMo scanning resp. detection algorithm

Post by Kyle_Katarn » Wed May 08, 2019 5:40 pm

In, at design time, I have all debugging tools I need. In order to support troubleshooting with end users, the current log file implementation is perfectly fine, in particular with debug mode turned off by default.

Post Reply