Wednesday, December 26, 2018

JAVA - Calculate distance between two wells' coordinates for Pad calculation

This is my first attempt to write JAVA code. could be better and better :)
This code loop through each one of the API in one file to 
get the distance between each one of them.


package com.nan.lens.wellcost;

import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.*;

import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVRecord;

public class Pad {

    private APIProvider provider;

    public Pad(APIProvider apiProvider) {
        this.provider = apiProvider;
    }

    public static Pad create() {
        try {
            return new Pad(new APIProvider("/apiFile.csv"));
        } catch (IOException e) {
            /*The throw statement creates a new object*/            
            throw new RuntimeException(e);
        }
    }

    // read data into java from csv    
     public static class APIProvider {
        // create map to store API_coordinates pair        
        private HashMap<String, ArrayList<Double>> API_LAT_LONG_1 = new HashMap<>();
        private HashMap<String, ArrayList<Double>> API_LAT_LONG_2;
        // create list to store coordinates        
        private ArrayList<Double> LAT_LONG = new ArrayList<Double>();
        private ArrayList<Double> temp = new ArrayList<Double>();

        public APIProvider(String apiFile) throws IOException {

            InputStream input = Pad.class.getResourceAsStream(apiFile);
            Iterable<CSVRecord> records = CSVFormat.EXCEL.parse(new InputStreamReader(input));

            for (CSVRecord record : records) {
                String key = record.get(0);
                temp.clear();
                /*Force to change type String to Double*/                
                temp.add(0, Double.valueOf(record.get(1)));
                temp.add(1, Double.valueOf(record.get(2)));
                LAT_LONG = (ArrayList<Double>) temp.clone();
                API_LAT_LONG_1.put(key, LAT_LONG);
            }
            //System.out.println("API_LAT_LONG: " + API_LAT_LONG_1);            
            API_LAT_LONG_2 = (HashMap) API_LAT_LONG_1.clone();

            HashMap<String, ArrayList<String>> API_DIST = new HashMap<>();
            ArrayList<String> api_dist_list;
            ArrayList<String> temp_list = new ArrayList<String>();
            final int R = 6371; // Radius of the earth in km
            for (String key : API_LAT_LONG_1.keySet()) {
                //get lat and long from ArrayList                
                List<Double> list = new ArrayList<Double>();
                list = API_LAT_LONG_1.get(key);
                Double lat1 = list.get(0);
                Double lng1 = list.get(1);

                for (String key2 : API_LAT_LONG_2.keySet()) {
                    List<Double> list2 = new ArrayList<Double>();
                    list2 = API_LAT_LONG_2.get(key2);
                    Double lat2 = list2.get(0);
                    Double lng2 = list2.get(1);

                    // Calculate distance between two points                    
                    double latDistance = Math.toRadians(lat2 - lat1);
                    double lonDistance = Math.toRadians(lng2 - lng1);
                    double a = Math.sin(latDistance / 2) * Math.sin(latDistance / 2)
                            + Math.cos(Math.toRadians(lat1)) * Math.cos(Math.toRadians(lat2))
                            * Math.sin(lonDistance / 2) * Math.sin(lonDistance / 2);
                    double c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
                    double distance = R * c; //km
                    distance = Math.pow(distance, 2);

                    if (key != key2 && distance < 0.0762) { // (km)                        
                        String UID = UUID.randomUUID().toString();
                        temp_list.clear();

                        temp_list.add(0, key);
                        temp_list.add(1, key2);
                        temp_list.add(2, String.valueOf(distance));

                        api_dist_list = (ArrayList<String>) temp_list.clone();
                        API_DIST.put(UID, api_dist_list);
                    }
                }
            }
            System.out.println(API_DIST);

        }
    }
}

Tuesday, December 4, 2018

Apache Kafka for beginners

https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html#


Apache Kafka and server concepts

Here are important concepts that you need to remember before we dig deeper into Apache Kafka - explained in one line.

  • Producer: Application that sends the messages.
  • Consumer: Application that receives the messages.
  • Message: Information that is sent from the producer to a consumer through Apache Kafka.
  • Connection: A connection is a TCP connection between your application and the Kafka broker.
  • Topic: A Topic is a category/feed name to which messages are stored and published.
  • Topic partition: Kafka topics are divided into a number of partitions, which allows you to split data across multiple brokers.
  • Replicas A replica of a partition is a "backup" of a partition. Replicas never read or write data. They are used to prevent data loss.
  • Consumer Group: A consumer group includes the set of consumer processes that are subscribing to a specific topic.
  • Offset: The offset is a unique identifier of a record within a partition. It denotes the position of the consumer in the partition.
  • Node: A node is a single computer in the Apache Kafka cluster.
  • Cluster: A cluster is a group of nodes i.e., a group of computers.

Thursday, August 23, 2018

SQL - drop all temp tables

declare @sql nvarchar(max)
select @sql = isnull(@sql+';', '') + 'drop table ' + quotename(name)
from tempdb..sysobjects
where name like '#[^#]%'
exec (@sql)

Tuesday, July 31, 2018

SQL - change exponential data to number

SELECT api, CONVERT(numeric(14,0), CAST(api AS FLOAT)) as new_api
FROM [dbo].[tbl_PA_Perforated]
where api like '%e%'



Python - multipart/form-data post requests

multipart/form-data contains boundary to separate name/value pairs. The boundary acts like a marker of each chunk of name/value pairs passed when a form gets submitted. The boundary is automatically added to a content-type of a request header.
The form with enctype="multipart/form-data" attribute will have a request header Content-Type : multipart/form-data; boundary --- WebKit193844043-h (browser generated vaue).
The payload passed looks something like this:
Content-Type: multipart/form-data; boundary=—-WebKitFormBoundary7MA4YWxkTrZu0gW

    --—-WebKitFormBoundary7MA4YWxkTrZu0gW
    Content-Disposition: form-data; name=”file”; filename=”captcha
    Content-Type:

    --—-WebKitFormBoundary7MA4YWxkTrZu0gW
    Content-Disposition: form-data; name=”action

    submit
    --—-WebKitFormBoundary7MA4YWxkTrZu0gW--
e.g.

from requests_toolbelt import MultipartEncoder
......

params = {
    "__EVENTVALIDATION": event_validation,
    "__VIEWSTATE": view_state,
    "MainContent_ScriptManager1_HiddenField": decode_content,
    "AutoCompleteCASNumbers": auto_CASnumber,
    "__VIEWSTATEGENERATOR": view_stategenerator,
    "ctl00$MainContent$cboHydSub": 'Submitted',
    "ctl00$MainContent$ddlRangeType": "Between",
    "ctl00$MainContent$tbRangeStartDate": fromDate,
    "ctl00$MainContent$tbRangeEndDate": toDate,
......}
url = 'https://fracfocusdata.org/DisclosureSearch/Search.aspx'
m = MultipartEncoder(params)
headers['Content-Type'] = m.content_type
resp = requests.post(url,headers=headers,data=m)
........



Reference:

https://www.jianshu.com/p/902452189ca9

Tuesday, July 24, 2018

SQL - Try_parse()

TRY_PARSE does two things - parse text using a specific culture and return NULL if the cast fails.


e.g.


with tbl as(
  select distinct api, ltrim(rtrim([GL/Ground Level/Elevation_above_MSL])) as GL, ltrim(rtrim(kb)) as KB
  FROM .[dbo].[tbl_NAWAT_COMPL_DirectionlSurvey] with (nolock)
  where kb is not null and kb <> 'NULL' and api is not null
 )

 select API, GL, TRY_PARSE(KB as float) as KB
 from(
select api, GL,
case when right(KB,4) = 'feet' then ltrim(rtrim(left(KB,len(KB)-4)))
when replace(KB,' ','') like '%@%usft%' then  left(right(KB, len(KB)-charindex('@',KB)),charindex('usft',right(KB, len(KB)-charindex('@',KB)-1)))
else KB
end as KB
from tbl
)d


Here I use try_parse because I only have two cases, but there are also characters beyond these two cases. since try_parse will return null for failed cases, I don't need to worry other cases.