NMS-16987: Initial Set of Data to Collect: Usage Stats #7573

smunir-onms · 2024-12-31T14:37:22Z

All Contributors

Have you read our Contribution Guidelines?
Have you (electronically) signed the OpenNMS Contributor Agreement?

Contribution Checklist

Please make an issue in the OpenNMS issue tracker if there isn't one already.
Once there is an issue, please:
1. update the title of this PR to be in the format: ${JIRA-ISSUE-NUMBER}: subject of pull request
2. update the Jira link at the bottom of this comment to refer to the real issue number
3. prefix your commit messages with the issue number, if possible
4. once you've created this PR, please link to it in a comment in the Jira issue
  Don't worry if this sounds like a lot, we can help you get things set up properly.
If this code is likely to affect the UI, did you name your branch with -smoke in it to trigger smoke tests?
If this is a new or updated feature, is there documentation for the new behavior?
If this is new code, are there unit and/or integration tests?
If this PR targets a foundation-* branch, does it try to avoid changing files in $OPENNMS_HOME/etc/?

What's Next?

A PR should be assigned at least 2 reviewers. If you know that someone would be a good person to review your code, feel free to add them.

If you need help making additions or changes to the documentation related to your changes, please let us know.

In any case, if anything is unclear or you want help getting your PR ready for merge, please don't hesitate to say something in the comments here,
or in the #opennms-development chat channel.

Once reviewer(s) accept the PR and the branch passes continuous integration, the PR is eligible for merge.

At that time, if you have commit access (are an OpenNMS Group employee or a member of the OGP) you are welcome to merge the PR when you're ready.
Otherwise, a reviewer can merge it for you.

Thanks for taking time to contribute!

External References

https://opennms.atlassian.net/browse/NMS-16987

…kip ci]

…e so [skip ci]

…ip ci]

…[skip ci]

cgorantla · 2025-01-16T15:15:28Z

...curity/src/main/java/org/opennms/web/springframework/security/OpenNMSAuthSuccessHandler.java

@@ -56,6 +56,11 @@ public void onAuthenticationSuccess(HttpServletRequest request, HttpServletRespo
        // check for admin/admin
        boolean defaultAdminLogin = isDefaultAdminLogin(request.getParameter("j_username"), request.getParameter("j_password"));

+        String ipAddress = getClientIpAddress(request);


remove csv logging from here.

Implement EventListener in Datachoices to capture the authlogin event and persist csv file from there.

Take reference from OpenNMSKafkaProducer for implementing this

mwajahatabbasi · 2025-01-27T13:18:19Z

...oices/src/main/java/org/opennms/features/datachoices/internal/usagestatistics/CsvLogger.java

+
+            // Append the new log entry to the CSV file
+            try (FileWriter writer = new FileWriter(csvFile, true)) {
+                String timestamp = LocalDateTime.now().format(TIMESTAMP_FORMATTER);


@smunir-onms The time should be captured from the login event rather than from where you're logging the information.

mwajahatabbasi · 2025-01-27T13:23:42Z

...oices/src/main/java/org/opennms/features/datachoices/internal/usagestatistics/CsvLogger.java

+    private static final String[] HEADER = {"Username", "Timestamp"};
+    private static final DateTimeFormatter TIMESTAMP_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
+
+    public static void logToCsv(String username) {


@smunir-onms, @cgorantla You can also use apache-commons-csv api for CSV manipulation.

mwajahatabbasi

LGTM.
@cgorantla , @smunir-onms As a suggestion, if the CSV file is only being used within Java, it would be better to consider using object serialization instead of a plain CSV file. This approach would enhance data security.

cgorantla · 2025-01-29T22:45:46Z

.../java/org/opennms/features/datachoices/internal/usagestatistics/UsageStatisticsReporter.java

@@ -348,6 +376,10 @@ private void setOpenNmsJmxAttributes(UsageStatisticsReportDTO usageStatisticsRep
        if (coreFlowsPersistedObj != null) {
            usageStatisticsReport.setCoreFlowsPersisted((long) coreFlowsPersistedObj);
        }
+        Object flowsPerSecondObj = getJmxAttribute(JMX_OBJ_OPENNMS_FLOWS_PER_SECOND, JMX_ATTR_COUNT);


I don't see any metric that we expose called flowsPerSecond.
We may have to expose this explicitly then we can collect this.

Also, you may want to test anything that you add.
Flows can be generated from here : https://github.com/OpenNMS/udpgen

@cgorantla Flows Per Second stuff removed from this PR, a new ticket is created to handle that. Please review other additional items. Thank you.

cgorantla · 2025-01-29T22:49:52Z

ui/src/components/UsageStatistics/UsageStatisticsTable.vue

+    console.error('No event found to download a csv.')
+    errorSnackbar.value = true
+    return
+  }
  // Decode the Base64 string into a byte array
  const byteCharacters = atob(statistics.value[item.key])
  const byteNumbers = new Array(byteCharacters.length).fill(null).map((_, i) => byteCharacters.charCodeAt(i))
  const byteArray = new Uint8Array(byteNumbers)

-  const fileType = metadata.value.metadata.find((meta) => meta.key === item.key)?.datatype.split('|').pop()?.toLowerCase()
-  const fileName = `Login events Past 60 days.${fileType}`
-  const contentType = fileType === 'csv' ? 'text/csv' : 'application/octet-stream'
+  const fileName = 'Login events Past 60 days.csv'

  // Create a Blob object for the CSV
-  const blob = new Blob([byteArray], { type: contentType})
+  const blob = new Blob([byteArray], { type: 'text/csv'})



may want to cleanup this to remove references to logins here.

Right, @Shahbaz-dataq was supposed to clean it up

cgorantla · 2025-02-13T17:31:44Z

opennms-base-assembly/src/main/filtered/etc/jmx-datacollection-config.xml

@@ -8,6 +8,9 @@
         <rra>RRA:MIN:0.5:288:366</rra>
      </rrd>
      <mbeans>
+         <mbean name="CPU Metrics" objectname="java.lang:type=OperatingSystem">


Do you need this here ?

Maybe some more CPU metrics will be added in the future, or this can be accommodated with other mbeans having type: OperatingSystem, please suggest. Thank you

Since this change is not needed for Usage stats, please remove it.

cgorantla · 2025-02-13T17:31:54Z

features/events/api/src/main/java/org/opennms/netmgt/events/api/EventConstants.java

+    /**
+     * login success UEI
+     */
+    public static final String AUTHENTICATION_SUCCESS_UEI = "uei.opennms.org/internal/authentication/successfulLogin";


don't need this here.

cgorantla · 2025-02-13T17:33:24Z

.../java/org/opennms/features/datachoices/internal/usagestatistics/UsageStatisticsReporter.java

@@ -268,7 +269,9 @@ public UsageStatisticsReportDTO generateReport() {
        usageStatisticsReport.setSnmpInterfacesWithFlows(m_snmpInterfaceDao.getNumInterfacesWithFlows());
        usageStatisticsReport.setMonitoredServices(m_monitoredServiceDao.countAll());
        usageStatisticsReport.setEvents(m_eventDao.countAll());
+        usageStatisticsReport.setEventsLastHours(m_eventDao.getNumEventsLastHours(24));


looks like 24 hrs is always constant. not configurable.

…ts/alarms

smunir-onms · 2025-02-17T12:48:19Z

opennms-base-assembly/src/main/filtered/etc/opennms.properties

+opennms.usage.stats.events.generated.last.hours=24
+opennms.usage.stats.alarms.generated.last.hours=24


@cgorantla is that fine? or should move to datachoices with a new properties file

You don't need to configure this here, put default property value in the code. Add a doc update on how to use this in data choices documentation.

smunir-onms added 6 commits December 13, 2024 19:27

NMS-16987, initial commit for new branch so [skip ci]

40a30a3

NMS-16987, added new items for alarms and events, branch update so [s…

f311b63

…kip ci]

NMS-16987, changes for flows and system cpu utilization, branch updat…

ebaf3b9

…e so [skip ci]

NMS-16987, completed system cpu/mem utilization, branch update so [sk…

d85e66f

…ip ci]

Merge branch 'develop' into smq/jira/NMS-16987

c0f089b

NMS-16987, CPU/MEM utilization testing validations, branch update so …

be61b58

…[skip ci]

smunir-onms requested review from Junaid-Khan-Nant, aurang-zeb313 and Mohammad-Junid December 31, 2024 14:37

smunir-onms added 3 commits December 31, 2024 19:40

NMS-16987, extra change reverted

09f700b

NMS-16987, added csv logger to keep login logs

ed4d6cf

NMS-16987, updated related test to fix build failure

95354aa

smunir-onms requested a review from naeem-afzal January 9, 2025 10:13

indigo423 changed the title ~~NMS-16987 Initial Set of Data to Collect: Usage Stats~~ NMS-16987: Initial Set of Data to Collect: Usage Stats Jan 9, 2025

smunir-onms marked this pull request as ready for review January 9, 2025 18:37

smunir-onms added 2 commits January 14, 2025 22:31

NMS-16987, added usage stats item for Login events csv download

d1d1e36

Merge branch 'refs/heads/develop' into smq/jira/NMS-16987

612ba60

cgorantla reviewed Jan 16, 2025

View reviewed changes

smunir-onms and others added 10 commits January 16, 2025 22:20

NMS-16987, added usage stats item for Login events csv download

d7f4868

NMS-16987, CSVLoger moved to datachoices

acbccea

NMS-16987, temporary base64 string for UI

48a42e1

NMS-16987, test case updated

1b4e426

Merge branch 'refs/heads/develop' into smq/jira/NMS-16987

27c5000

NMS-16987, CSVLogger functionality moved to datachoices

a763789

NMS-16987, updated tests to for build issue

71824b9

NMS-16987, changes to fix smoke test

1b7b298

fix for empty file

a6d1fe4

lint fixes

ff5bcc5

smunir-onms requested review from cgorantla and mwajahatabbasi January 23, 2025 12:17

mwajahatabbasi reviewed Jan 27, 2025

View reviewed changes

smunir-onms added 4 commits January 28, 2025 16:07

NMS-16987, login timestamp from authevent

1f24b6d

NMS-16987, removed login logs csv stuf from usage stats

6b4d88a

Merge branch 'develop' into smq/jira/NMS-16987

5066702

NMS-16987, removed login logs csv and listenr classes from this PR

8022d24

cgorantla reviewed Jan 29, 2025

View reviewed changes

Shahbaz-dataq and others added 8 commits January 30, 2025 19:22

changes reverted for csv download

2d13de6

NMS-16987, Flows per second item removed in this branch, might be needed

b2d7fa7

Merge branch 'develop' into smq/jira/NMS-16987

eefcf1d

Merge branch 'develop' into smq/jira/NMS-16987

d6a8bd8

Merge branch 'develop' into smq/jira/NMS-16987

c9db179

Merge branch 'develop' into smq/jira/NMS-16987-without-fps

64bf68d

Merge branch 'smq/jira/NMS-16987-without-fps' into smq/jira/NMS-16987

ebe55fb

NMS-16987, Flows per second item removed from usage stats

901f85c

smunir-onms requested review from mwajahatabbasi and cgorantla February 11, 2025 16:07

cgorantla reviewed Feb 13, 2025

View reviewed changes

smunir-onms added 3 commits February 14, 2025 20:23

Merge branch 'refs/heads/develop' into smq/jira/NMS-16987

4befa14

NMS-16987, code review updates, last hours made configurable for even…

d376fb3

…ts/alarms

NMS-16987, metadata update for configured values

d9769d8

smunir-onms commented Feb 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NMS-16987: Initial Set of Data to Collect: Usage Stats #7573

NMS-16987: Initial Set of Data to Collect: Usage Stats #7573

smunir-onms commented Dec 31, 2024

cgorantla Jan 16, 2025

cgorantla Jan 20, 2025

mwajahatabbasi Jan 27, 2025

mwajahatabbasi Jan 27, 2025 •

edited

Loading

mwajahatabbasi left a comment •

edited

Loading

cgorantla Jan 29, 2025

smunir-onms Feb 11, 2025

cgorantla Jan 29, 2025

smunir-onms Jan 30, 2025

cgorantla Feb 13, 2025

smunir-onms Feb 14, 2025 •

edited

Loading

cgorantla Feb 18, 2025

cgorantla Feb 13, 2025

smunir-onms Feb 14, 2025

cgorantla Feb 13, 2025

smunir-onms Feb 14, 2025

smunir-onms Feb 17, 2025

cgorantla Feb 18, 2025

		opennms.usage.stats.events.generated.last.hours=24
		opennms.usage.stats.alarms.generated.last.hours=24

NMS-16987: Initial Set of Data to Collect: Usage Stats #7573

Are you sure you want to change the base?

NMS-16987: Initial Set of Data to Collect: Usage Stats #7573

Conversation

smunir-onms commented Dec 31, 2024

All Contributors

Contribution Checklist

What's Next?

External References

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwajahatabbasi Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

mwajahatabbasi left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smunir-onms Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwajahatabbasi Jan 27, 2025 •

edited

Loading

mwajahatabbasi left a comment •

edited

Loading

smunir-onms Feb 14, 2025 •

edited

Loading